Study status: 200/200 stages have reached a reported outcome; 193 returned a final response and 200 have a source-based grade. The full matrix is evaluated.
We gave five models the same work: reconcile a messy month of sales, choose a laptop, repair a small website, recover project decisions, write from a source packet, diagnose an export problem, triage customer tickets, schedule a meeting, check invoice images and extract current requirements from a long archive. Each task was run twice. Every initial submission then received the same self-review request, without the answer key.
The question was practical: what can someone use, what needs fixing, and what did it cost to get there? This is an inspection of returned work, not a claim that a model can run a company. Every business, customer, offer, invoice and software screenshot in the test is a constructed fixture. The tasks resemble real work while giving us known facts to check.
Open all 200 result rows and their files. Download the exact candidate inputs. Read the grading rubric. The downloadable guide-and-evidence bundle keeps these links together: extract it before opening the guide or evidence index.
A result meets our usability threshold only if it scores at least 16/20, scores at least 3/4 on each criterion, and has no defined critical failure. “Minor edits” meets that threshold; “major repair” does not. Five strong totals cannot compensate for one missing essential deliverable.
| Model | Initial usable / graded | Initial mean /20 | Revised usable / graded | Revised mean /20 | Median initial minutes | Receipted study cost, USD |
|---|---|---|---|---|---|---|
| Qwen4-Max | 18/20 | 18.65 | 18/20 | 18.80 | 2.69 | 3.6174 |
| GPT-6 Astra | 20/20 | 19.80 | 20/20 | 19.85 | 1.28 | 17.4133 |
| GPT-6.1 Sol | 19/20 | 19.60 | 20/20 | 19.90 | 1.36 | 2.8383 |
| Claude Opus 5.5 | 18/20 | 19.20 | 18/20 | 19.45 | 1.47 | 23.9202 |
| Claude Fable 5.2 | 16/20 | 17.30 | 17/20 | 18.50 | 2.07 | 61.3015 |
Qwen4-Max: among 20 graded pairs, scores rose in 4, stayed equal in 15 and fell in 1. 0 crossed into the usable range; 0 crossed out.
GPT-6 Astra: among 20 graded pairs, scores rose in 1, stayed equal in 19 and fell in 0. 0 crossed into the usable range; 0 crossed out.
GPT-6.1 Sol: among 20 graded pairs, scores rose in 4, stayed equal in 16 and fell in 0. 1 crossed into the usable range; 0 crossed out.
Claude Opus 5.5: among 20 graded pairs, scores rose in 6, stayed equal in 13 and fell in 1. 0 crossed into the usable range; 0 crossed out.
Claude Fable 5.2: among 20 graded pairs, scores rose in 5, stayed equal in 15 and fell in 0. 1 crossed into the usable range; 0 crossed out.
Astra cleared our usability threshold on all 20 initial submissions and all 20 revisions. Sol cleared it on 19 of 20 initials and all 20 revisions; its missing initial website handoff was the exception. Those are the strongest observed delivery records in this particular kit. They are not proof of superiority on unseen tasks, and the grading was not blinded.
Qwen cleared the threshold on 18 of 20 initial submissions and 18 of 20 revisions. Its receipted cost is shown above and is worth comparing with the specific repairs needed, especially the structured refund fields. Opus also cleared 18 of 20 in both stages; its support outputs show why a high average can coexist with a material weak criterion. Fable cleared 16 of 20 initials and 17 of 20 revisions, with three no-file truncations across the second accounting and coordination attempts. Its other completed outputs should still be judged on their own evidence.
These counts include genuine limit-hit outcomes graded as delivered. They do not turn historical provider interruptions into model failures. We would use this study to shortlist models for a particular workflow, then test fresh examples from that workflow before relying on it.
Qwen4-Max: 2 of 20 reviewed initial submissions earned full marks. Initial outputs below our usability threshold: task 7, repeat 1 (14/20), task 7, repeat 2 (17/20). The task tables below link the exact outputs and the separate revision outcomes.
GPT-6 Astra: 16 of 20 reviewed initial submissions earned full marks. Every reviewed initial output met the usability threshold. The task tables below link the exact outputs and the separate revision outcomes.
GPT-6.1 Sol: 14 of 20 reviewed initial submissions earned full marks. Initial outputs below our usability threshold: task 3, repeat 1 (17/20). The task tables below link the exact outputs and the separate revision outcomes.
Claude Opus 5.5: 10 of 20 reviewed initial submissions earned full marks. Initial outputs below our usability threshold: task 7, repeat 1 (16/20), task 7, repeat 2 (16/20). The task tables below link the exact outputs and the separate revision outcomes.
Claude Fable 5.2: 10 of 20 reviewed initial submissions earned full marks. Initial outputs below our usability threshold: task 1, repeat 2 (0/20), task 4, repeat 2 (0/20), task 7, repeat 1 (17/20), task 7, repeat 2 (16/20). The task tables below link the exact outputs and the separate revision outcomes.
The support queue is a useful caution about structured output. Several submissions entered numeric zero for an undecided refund while surrounding prose said the decision was unresolved. An imported CSV may lose those qualifications. The source record required unknown amounts to stay unknown; this is a material data-handling error, not a tone preference. The task-seven tables and source-linked reviews show exactly which models and repetitions did this.
The website task exposed a different weakness: spending the call budget on repeated checks can leave the handoff unfinished even when the app works. Conversely, reaching a turn limit does not prove the saved work is unusable. We inspect both the delivered files and the actual endpoint behavior. The initial and revision rows deliberately preserve these distinctions.
In the source-synthesis tasks, correct headline answers sometimes came with invented explanations: a conflict attributed to the wrong document, an owner assignment treated as settled, or an approval condition added to the brief. Small-sounding additions can change what a person does next. The reviews identify the unsupported passage and the correction rather than treating fluent language as evidence.
Fable's second accounting initial run exhausted the output allowance without saving the required deliverables. Its revision recovered them. On the second coordination run, both the initial and revision exhausted the allowance with no required files. These are observed delivery failures under the tested configuration, not proof that a different token budget or app would behave the same way. The accounting recovery and the coordination failure both remain visible.
Read each numerator together with its denominator. These averages describe ten chosen fixtures with two repetitions, not a statistically representative population of jobs. A one-point editorial difference is less informative than the linked error. Costs include locally receipted earlier attempts; the unresolved historical aggregate difference cannot honestly be assigned to a model. Timings span recovery waves and changing load, so they are observations rather than a controlled speed ranking.
There were 100 initial runs: ten tasks × five models × two fresh repetitions. Each had one fixed revision stage, producing 200 planned outcomes. The second repetition uses the same fixture; it checks repeatability on that fixture, not generalization to unseen examples. A revision may recover a failed delivery, but its score never replaces the first score.
| Family | Exact requested model | Pinned provider | Reasoning setting |
|---|---|---|---|
| Qwen 4 | qwen/qwen4-max |
Alibaba | high |
| Astra | openai/gpt-6-astra |
OpenAI | medium |
| Sol | openai/gpt-6.1-sol |
OpenAI | medium |
| Opus | anthropic/claude-opus-5.5 |
Anthropic | high |
| Fable | anthropic/claude-fable-5.2 |
Anthropic | high |
These are OpenRouter API tests with provider fallback disabled, not tests of the vendors' chat apps. We explicitly selected the reasoning defaults observed in the catalog. Equal prompts and tools do not make these equal-compute configurations. Saved catalog and endpoint snapshots establish the identifiers and pricing available at execution.
Every candidate received the shared instructions and only its active task. Available tools could list, read and write local files, execute standard-library Python and inspect a local browser. Candidate tools had no external network access and could not read credentials, grading keys or other runs. Candidate Python also could not bind a local socket; the browser tool supplied the documented local API instead. Models attempting to launch their own server received a permission error and had to use the available tools. We preserve those errors because this environment constraint influenced the work. Text and CSV contents were supplied directly; image tasks received the same PNG inputs. The browser returned text, controls and overflow measurements, not screenshots.
Each stage had up to 16 model calls, with 16,384 output tokens per call including reasoning, and a nominal 30-minute initial or 15-minute revision allowance. A request already in flight could finish after that time. Early tasks had a 220,000-byte request cap; later tasks and recovery runs used 800,000 bytes. Initial batches ran three jobs concurrently; recovery used two and streamed responses to preserve generation IDs early. These changes limit timing comparisons. The exact history is in method corrections and the execution settings.
Infrastructure interruptions were retried in separately identified recovery attempts. Missing initial work restarted from the task inputs. Missing revisions resumed the saved initial conversation and restored the initial output folder; incidental workspace scratch files could remain. This is a disclosed recovery limitation. Genuine turn limits and output truncation were retained as outcomes rather than retried for a better score.
Each task has five criteria worth 0–4 points. The answer keys existed outside candidate workspaces. A separate AI reviewer checked output against the source packet and recorded criterion-specific evidence, issues and file hashes. The reviewer could see model identities: this was unblinded AI review, not a blinded human panel. The operator also performed deterministic checks and actual website integration tests. Claims about correctness rest on source reconciliation and those checks, not on model agreement alone.
For the website, the candidate browser tool served a trusted implementation of the supplied lead API. That let models inspect frontend behavior without exposing an arbitrary server. Separately, the operator ran each delivered Python server in a disposable copy, checked persistence and invalid requests, verified prices, and captured 390- and 1,280-pixel views. A page can work while its required handoff is still missing; those receive separate credit.
We did not measure human checking or repair time. A model's estimate of time saved is not a measurement and is excluded. “Complete” in an API log means the response ended; it does not mean the work was correct. The grading sheets preserve that distinction.
Tables show both repetitions and both stages. Each cell gives the grade, usability category and execution status where it was not a normal completion. The time and cost columns show initial → revision for the selected attempts. All earlier attempts and their charges are linked under Files.
Assignment: Reconcile sales, refunds and advertising.
The checkable result is nine orders, CAD 2,820 gross sales, CAD 300 in applied refunds and CAD 450 in advertising. Net revenue of CAD 2,520 and contribution after ads of CAD 2,070 remain provisional because a settled Course refund has no amount. Workshop totals are unaffected. Another refund has no matching order and must remain unallocated. Contribution after ads is not business profit.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 18/20; minor edits | 18/20; minor edits | 3.66 → 2.40 | 0.12818 → 0.09011 | Files |
| Qwen4-Max | 2 | 19/20; minor edits | 19/20; minor edits | 3.63 → 3.32 | 0.13885 → 0.11890 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable | 1.70 → 1.96 | 0.40114 → 0.53620 | Files |
| GPT-6 Astra | 2 | 20/20; usable | 20/20; usable | 1.71 → 1.46 | 0.40243 → 0.67919 | Files |
| GPT-6.1 Sol | 1 | 20/20; usable | 20/20; usable | 1.45 → 1.53 | 0.07905 → 0.09904 | Files |
| GPT-6.1 Sol | 2 | 20/20; usable | 20/20; usable | 1.93 → 0.88 | 0.07704 → 0.09244 | Files |
| Claude Opus 5.5 | 1 | 19/20; minor edits | 20/20; usable | 1.61 → 1.69 | 0.61298 → 0.80026 | Files |
| Claude Opus 5.5 | 2 | 20/20; usable | 19/20; minor edits | 1.75 → 1.26 | 0.41643 → 0.92980 | Files |
| Claude Fable 5.2 | 1 | 19/20; minor edits | 20/20; usable | 3.38 → 2.13 | 2.20970 → 2.34745 | Files |
| Claude Fable 5.2 | 2 | 0/20; failed; output_truncated | 20/20; usable | 2.33 → 2.86 | 0.68170 → 2.63500 | Files |
What to inspect: Qwen4-Max, repeat 1: RF04 is currently excluded but still requires reconciliation and could change final results. (output) Claude Opus 5.5, repeat 1: Refund on an unknown order; N9999 is absent. (output) Claude Fable 5.2, repeat 1: Only ad spend is assigned to products; sales-to-ad attribution is not given. (output)
Assignment: Choose a laptop within a landed-cost and delivery budget.
Only offer O2 meets the stated requirements under the scenario GPU interpretation. Its landed price is CAD 2,098.88. The challenge is reconciling price, specifications, tax, shipping and delivery evidence; the offers are fictional and this is not a live buying recommendation.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 19/20; minor edits | 19/20; minor edits | 1.47 → 2.24 | 0.03793 → 0.07512 | Files |
| Qwen4-Max | 2 | 19/20; minor edits | 20/20; usable | 2.35 → 1.57 | 0.07988 → 0.06871 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable | 0.81 → 1.22 | 0.21863 → 0.35603 | Files |
| GPT-6 Astra | 2 | 20/20; usable | 20/20; usable | 0.93 → 1.00 | 0.22004 → 0.32120 | Files |
| GPT-6.1 Sol | 1 | 20/20; usable | 20/20; usable | 0.76 → 0.93 | 0.05031 → 0.06249 | Files |
| GPT-6.1 Sol | 2 | 20/20; usable | 20/20; usable | 1.77 → 2.11 | 0.04680 → 0.06424 | Files |
| Claude Opus 5.5 | 1 | 20/20; usable | 20/20; usable | 1.07 → 1.11 | 0.27598 → 0.47794 | Files |
| Claude Opus 5.5 | 2 | 20/20; usable | 20/20; usable | 1.19 → 1.58 | 0.25583 → 0.85035 | Files |
| Claude Fable 5.2 | 1 | 20/20; usable | 20/20; usable | 1.41 → 2.41 | 0.65512 → 1.74429 | Files |
| Claude Fable 5.2 | 2 | 20/20; usable | 20/20; usable | 1.33 → 1.85 | 0.60769 → 1.60013 | Files |
What to inspect: Qwen4-Max, repeat 1: O5 has a differently named GPU and no supplied performance test; omit comparative strength unless source verifies it. (output) Claude Opus 5.5, repeat 1: Tied at 2.3kg with the other Forge SKU; P16-32 is heavier at 2.5kg. Does not affect purchase decision. (output)
Assignment: Repair signup persistence, pricing and mobile layout.
The page posts to the wrong endpoint, applies the discount incorrectly and overflows on mobile. The checks run the delivered server and verify real saved records, invalid requests, four totals and two viewport widths. A working site and a complete handoff are scored separately.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 19/20; minor edits | 19/20; minor edits | 4.55 → 4.07 | 0.12196 → 0.12975 | Files |
| Qwen4-Max | 2 | 20/20; usable; turn_limit | 20/20; usable | 4.27 → 3.28 | 0.13952 → 0.14781 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable; turn_limit | 3.16 → 2.00 | 0.81461 → 1.14669 | Files |
| GPT-6 Astra | 2 | 20/20; usable | 20/20; usable | 2.29 → 2.50 | 0.64110 → 0.90385 | Files |
| GPT-6.1 Sol | 1 | 17/20; major repair; turn_limit | 20/20; usable | 1.91 → 1.95 | 0.08201 → 0.08290 | Files |
| GPT-6.1 Sol | 2 | 20/20; usable; turn_limit | 20/20; usable | 2.12 → 1.55 | 0.12813 → 0.12517 | Files |
| Claude Opus 5.5 | 1 | 20/20; usable | 20/20; usable | 2.66 → 2.66 | 0.95239 → 2.11108 | Files |
| Claude Opus 5.5 | 2 | 20/20; usable | 20/20; usable | 2.53 → 1.86 | 1.13494 → 1.40546 | Files |
| Claude Fable 5.2 | 1 | 20/20; usable | 20/20; usable | 4.02 → 3.12 | 2.56330 → 2.67105 | Files |
| Claude Fable 5.2 | 2 | 20/20; usable | 20/20; usable | 4.16 → 4.10 | 3.14998 → 4.63975 | Files |
What to inspect: Qwen4-Max, repeat 1: Original supplied blank/0/fractional/out-of-range numeric inputs produce wrong numeric totals; NaN case not evidenced. (output) GPT-6.1 Sol, repeat 1: Supply handoff/run instructions/check evidence and remove fake submissions from runnable package. (output) Claude Opus 5.5, repeat 2: toggleAttribute aria-invalid yields empty attribute rather than literal true. Does not change required validation behavior, but use setAttribute true/false for accessibility cleanup. (output)
Assignment: Resolve old decisions and a later approved change.
The approved event is October 30 at 11:00 Pacific, for 80 people, free, on MeetRoom. Recording needs consent and paid ads remain on hold. Unassigned testing and recipient permission must remain open rather than be invented.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 18/20; minor edits | 19/20; minor edits | 2.67 → 2.34 | 0.07216 → 0.11110 | Files |
| Qwen4-Max | 2 | 19/20; minor edits | 19/20; minor edits | 2.00 → 2.69 | 0.06173 → 0.12070 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable | 2.25 → 2.02 | 0.38701 → 0.46338 | Files |
| GPT-6 Astra | 2 | 19/20; minor edits | 19/20; minor edits | 1.86 → 1.59 | 0.38488 → 0.57925 | Files |
| GPT-6.1 Sol | 1 | 19/20; minor edits | 20/20; usable | 2.06 → 2.09 | 0.05514 → 0.09999 | Files |
| GPT-6.1 Sol | 2 | 19/20; minor edits | 19/20; minor edits | 1.46 → 1.27 | 0.06541 → 0.09074 | Files |
| Claude Opus 5.5 | 1 | 19/20; minor edits | 19/20; minor edits | 1.76 → 2.11 | 0.41660 → 1.22648 | Files |
| Claude Opus 5.5 | 2 | 19/20; minor edits | 19/20; minor edits | 2.05 → 1.37 | 0.80459 → 0.67347 | Files |
| Claude Fable 5.2 | 1 | 19/20; minor edits | 20/20; usable | 3.26 → 3.29 | 1.28666 → 1.59423 | Files |
| Claude Fable 5.2 | 2 | 0/20; failed; output_truncated | 0/20; failed; output_truncated | 2.49 → 2.21 | 0.67831 → 1.01057 | Files |
What to inspect: Qwen4-Max, repeat 1: Campaign approver unspecified; proposed Priya if desired, not confirmed. (output) GPT-6 Astra, repeat 2: Recipient-scope decision is delegated to an unassigned verifier or decision-maker, although Priya is coordinator and Mei explicitly asks her to resolve it. Assign Priya accountable resolution; she may delegate verification. (output) GPT-6.1 Sol, repeat 1: Approver unspecified; Priya may be proposed coordinator but authority not established. (output)
Assignment: Write accurately from a fictional AI-release packet.
The supported price is USD 29 with three seats minimum. An erroneous launch screenshot must not override the corrected source. Vendor draft-time savings are not end-to-end savings. The article must not invent hands-on testing, customer quotes, privacy guarantees or certification.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 19/20; minor edits | 19/20; minor edits | 2.71 → 2.32 | 0.06942 → 0.07074 | Files |
| Qwen4-Max | 2 | 19/20; minor edits | 19/20; minor edits | 1.89 → 2.28 | 0.07234 → 0.08200 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable | 1.51 → 1.95 | 0.29422 → 0.52611 | Files |
| GPT-6 Astra | 2 | 20/20; usable | 20/20; usable | 1.31 → 1.55 | 0.28194 → 0.49826 | Files |
| GPT-6.1 Sol | 1 | 20/20; usable | 20/20; usable | 1.39 → 1.61 | 0.04998 → 0.06945 | Files |
| GPT-6.1 Sol | 2 | 20/20; usable | 20/20; usable | 1.30 → 0.76 | 0.05280 → 0.06014 | Files |
| Claude Opus 5.5 | 1 | 20/20; usable | 20/20; usable | 1.52 → 1.57 | 0.34373 → 0.56879 | Files |
| Claude Opus 5.5 | 2 | 20/20; usable | 20/20; usable | 1.19 → 1.15 | 0.28662 → 0.62060 | Files |
| Claude Fable 5.2 | 1 | 19/20; minor edits | 19/20; minor edits | 2.53 → 2.59 | 0.98804 → 1.74074 | Files |
| Claude Fable 5.2 | 2 | 18/20; minor edits | 19/20; minor edits | 1.85 → 2.31 | 0.78039 → 1.99872 | Files |
What to inspect: Qwen4-Max, repeat 1: The source provides waitlist access but no fee terms; say newcomers can join waitlist. (output) Claude Fable 5.2, repeat 1: API/EUaccess is planned and not currently available under source packet. (output)
Assignment: Diagnose export failure from three screenshots.
The blocking problem is offline media. Relink the original clip, preserve the edit, switch captions from a sidecar file to burned-in if they must appear in the MP4, and review the nonblocking audio warning. The model must not claim it operated the app.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 20/20; usable | 20/20; usable | 1.16 → 1.73 | 0.02760 → 0.03384 | Files |
| Qwen4-Max | 2 | 19/20; minor edits | 19/20; minor edits | 1.00 → 1.78 | 0.02665 → 0.04533 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable | 0.75 → 0.74 | 0.16037 → 0.14852 | Files |
| GPT-6 Astra | 2 | 19/20; minor edits | 19/20; minor edits | 0.53 → 0.67 | 0.15519 → 0.14549 | Files |
| GPT-6.1 Sol | 1 | 20/20; usable | 20/20; usable | 0.65 → 0.72 | 0.02895 → 0.02229 | Files |
| GPT-6.1 Sol | 2 | 19/20; minor edits | 19/20; minor edits | 0.43 → 0.76 | 0.02922 → 0.03155 | Files |
| Claude Opus 5.5 | 1 | 20/20; usable | 20/20; usable | 1.10 → 0.92 | 0.22033 → 0.25100 | Files |
| Claude Opus 5.5 | 2 | 19/20; minor edits | 19/20; minor edits | 0.76 → 1.27 | 0.21769 → 0.73012 | Files |
| Claude Fable 5.2 | 1 | 20/20; usable | 20/20; usable | 1.36 → 2.08 | 0.55947 → 0.99602 | Files |
| Claude Fable 5.2 | 2 | 19/20; minor edits | 19/20; minor edits | 1.11 → 2.06 | 0.60130 → 1.88536 | Files |
What to inspect: Qwen4-Max, repeat 2: Final verification should explicitly check the 8–24-second interview segment, subtitle timing/synchronization and audio around 42 seconds in exported MP4; current step 6 mentions only visible captions and interview plays. (output) GPT-6 Astra, repeat 2: Include explicit subtitle synchronization/timing check in resulting MP4; identify the 8–24-second review range clearly in final playback instruction. (output) GPT-6.1 Sol, repeat 2: Include explicit subtitle synchronization/timing check in resulting MP4; identify the 8–24-second review range clearly in final playback instruction. (output)
Assignment: Triage ten tickets against a store policy.
The queue tests return-window boundaries, a 20% course-viewing threshold, pending versus settled payments, a dangerous adapter and missing identity. An unknown refund is not numeric zero. Draft replies must not imply an action has already been performed.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 14/20; major repair | 15/20; major repair | 3.53 → 2.42 | 0.10451 → 0.10489 | Files |
| Qwen4-Max | 2 | 17/20; major repair | 17/20; major repair | 3.94 → 1.86 | 0.11445 → 0.06115 | Files |
| GPT-6 Astra | 1 | 19/20; minor edits | 20/20; usable | 1.45 → 1.38 | 0.35221 → 0.46533 | Files |
| GPT-6 Astra | 2 | 20/20; usable | 20/20; usable | 1.26 → 1.30 | 0.26660 → 0.36601 | Files |
| GPT-6.1 Sol | 1 | 20/20; usable | 20/20; usable | 1.18 → 0.85 | 0.05454 → 0.05393 | Files |
| GPT-6.1 Sol | 2 | 20/20; usable | 20/20; usable | 1.29 → 1.14 | 0.05308 → 0.05586 | Files |
| Claude Opus 5.5 | 1 | 16/20; major repair | 17/20; major repair | 1.05 → 0.71 | 0.21179 → 0.23762 | Files |
| Claude Opus 5.5 | 2 | 16/20; major repair | 17/20; major repair | 1.79 → 1.95 | 0.29066 → 0.70858 | Files |
| Claude Fable 5.2 | 1 | 17/20; major repair | 17/20; major repair | 2.07 → 1.94 | 0.83135 → 1.34273 | Files |
| Claude Fable 5.2 | 2 | 16/20; major repair | 16/20; major repair | 2.06 → 1.72 | 0.82314 → 1.00605 | Files |
What to inspect: Qwen4-Max, repeat 1: Source A704is2026-10-09,day11. (output) GPT-6 Astra, repeat 1: Acknowledge refund request without invented motive. (output) Claude Opus 5.5, repeat 1: Use blank/null for undetermined compensation and missing-order amount. (output)
Assignment: Find the earliest feasible meeting under resource constraints.
The earliest review is 14:00–14:45 Pacific in Room C, with preparation at 13:30, setup at 13:50 and follow-up at 14:45. The security decision remains unresolved. Availability, setup and follow-up all constrain the answer.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 19/20; minor edits | 19/20; minor edits | 2.46 → 2.66 | 0.06911 → 0.08976 | Files |
| Qwen4-Max | 2 | 19/20; minor edits | 19/20; minor edits | 2.24 → 2.54 | 0.07523 → 0.08128 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable | 1.17 → 1.16 | 0.25852 → 0.30926 | Files |
| GPT-6 Astra | 2 | 20/20; usable | 20/20; usable | 1.14 → 1.45 | 0.23179 → 0.32698 | Files |
| GPT-6.1 Sol | 1 | 20/20; usable | 20/20; usable | 0.99 → 0.92 | 0.04754 → 0.04634 | Files |
| GPT-6.1 Sol | 2 | 19/20; minor edits | 20/20; usable | 1.33 → 0.95 | 0.04744 → 0.03653 | Files |
| Claude Opus 5.5 | 1 | 19/20; minor edits | 20/20; usable | 1.19 → 1.32 | 0.27925 → 0.45598 | Files |
| Claude Opus 5.5 | 2 | 19/20; minor edits | 19/20; minor edits | 1.42 → 1.18 | 0.27214 → 0.29789 | Files |
| Claude Fable 5.2 | 1 | 20/20; usable | 20/20; usable | 2.10 → 2.01 | 0.86816 → 1.60974 | Files |
| Claude Fable 5.2 | 2 | 20/20; usable | 20/20; usable | 1.63 → 1.69 | 0.52714 → 1.06790 | Files |
What to inspect: Qwen4-Max, repeat 1: Room schedule is supplied; assistant availability is an assumption, not provided. (output) GPT-6.1 Sol, repeat 2: Separate identifying/requesting security approval from booking the otherwise feasible review. (output) Claude Opus 5.5, repeat 1: Keep ownership unresolved without asserting it blocks sending the scheduling invitation. (output)
Assignment: Extract and reconcile two invoices.
Invoice 001083 prints 329.20, but its lines and tax total 319.20. It bills three microphones against two authorized and changes the bank suffix from 8821 to 8891. Verification and a payment hold are appropriate; the evidence does not prove fraud.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 19/20; minor edits | 19/20; minor edits | 1.87 → 2.35 | 0.03853 → 0.04530 | Files |
| Qwen4-Max | 2 | 19/20; minor edits | 19/20; minor edits | 3.11 → 1.57 | 0.08034 → 0.05565 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable | 1.05 → 0.95 | 0.24485 → 0.24019 | Files |
| GPT-6 Astra | 2 | 20/20; usable | 20/20; usable | 1.29 → 0.68 | 0.18129 → 0.29208 | Files |
| GPT-6.1 Sol | 1 | 20/20; usable | 20/20; usable | 1.58 → 1.12 | 0.05403 → 0.03235 | Files |
| GPT-6.1 Sol | 2 | 20/20; usable | 20/20; usable | 0.84 → 0.72 | 0.05246 → 0.03943 | Files |
| Claude Opus 5.5 | 1 | 19/20; minor edits | 20/20; usable | 1.21 → 1.13 | 0.32314 → 0.38278 | Files |
| Claude Opus 5.5 | 2 | 20/20; usable | 20/20; usable | 0.96 → 0.59 | 0.20404 → 0.20940 | Files |
| Claude Fable 5.2 | 1 | 20/20; usable | 20/20; usable | 1.99 → 1.51 | 0.91500 → 0.90250 | Files |
| Claude Fable 5.2 | 2 | 20/20; usable | 20/20; usable | 1.52 → 1.34 | 0.59564 → 0.78603 | Files |
What to inspect: Qwen4-Max, repeat 1: Footer states shipping tax exemption; it does not authenticate shipping approval. Mark as evidence gap. (output) Claude Opus 5.5, repeat 1: Describe mismatch and trusted-contact verification; do not imply a supplied APprocedure or sourced prevalence claim. (output)
Assignment: Recover current scope and launch prerequisites.
Current scope excludes email alerts and SSO, limits the pilot to 80 accounts and requires an accessibility fix, rollback plan, written signoff and trained operator. A reserved deployment slot is not launch authorization. Instructions quoted from a customer remain source data rather than authority over the task.
| Model | Repeat | Initial | Revision | Minutes, initial → revised | USD, initial → revised | Evidence |
|---|---|---|---|---|---|---|
| Qwen4-Max | 1 | 19/20; minor edits | 18/20; minor edits | 2.72 → 2.05 | 0.07273 → 0.07151 | Files |
| Qwen4-Max | 2 | 19/20; minor edits | 20/20; usable | 3.22 → 2.22 | 0.07689 → 0.09260 | Files |
| GPT-6 Astra | 1 | 20/20; usable | 20/20; usable | 1.09 → 1.00 | 0.46915 → 0.64626 | Files |
| GPT-6 Astra | 2 | 19/20; minor edits | 19/20; minor edits | 1.07 → 1.19 | 0.47647 → 0.51443 | Files |
| GPT-6.1 Sol | 1 | 20/20; usable | 20/20; usable | 2.20 → 2.30 | 0.04378 → 0.08177 | Files |
| GPT-6.1 Sol | 2 | 19/20; minor edits | 20/20; usable | 1.08 → 1.16 | 0.08448 → 0.07813 | Files |
| Claude Opus 5.5 | 1 | 20/20; usable | 20/20; usable | 2.13 → 2.15 | 0.67461 → 0.98743 | Files |
| Claude Opus 5.5 | 2 | 19/20; minor edits | 20/20; usable | 1.87 → 1.76 | 0.42832 → 0.44180 | Files |
| Claude Fable 5.2 | 1 | 19/20; minor edits | 20/20; usable | 2.27 → 2.66 | 1.52367 → 1.98448 | Files |
| Claude Fable 5.2 | 2 | 20/20; usable | 20/20; usable | 1.68 → 2.26 | 0.92692 → 1.63811 | Files |
What to inspect: Qwen4-Max, repeat 1: Smoke executor, pilot-opening owner and reserved-slot owner unspecified; label inferred roles. (output) GPT-6 Astra, repeat 2: may deploy, no earlier than the conditional October 30 target (output) GPT-6.1 Sol, repeat 2: Asserts deployment cannot occur before October 30; source defines earliest pilot target, not separate prelaunch deployment prohibition. Keep target conditional without invented deployment-date limit. (output)
At the latest saved accounting check, cumulative inference is conservatively accounted at US$109.25905 against the authorized US$125 ceiling. This comprises US$67.5263644 in reconciled historical usage plus US$41.73269 in completion-pass receipts. Pending new-call reservations are US$0.00000. A reservation is a maximum allowance, not a confirmed charge.
The historical response receipts totaled US$67.3580344. The key's aggregate usage change was US$67.5263644, leaving US$0.16833 unassigned. That difference stays conservatively in the study accounting. Earlier unknown-call reservations totaled US$26.70488375; adding those to reconciled usage would double-count uncertainty as if it were confirmed spend. We preserve the original ledgers and distinguish them from reconciliation.
The approved top-up purchased US$50 in credits for a US$59.08 cash payment: US$50 credit principal, US$2.75 fee and US$6.33 tax. The credit purchase and later inference consumption are two views of the same funding, so they are not added together as model-test cost. Fees and tax are separately disclosed cash costs. Remaining credits are available account funds, not evidence that they were spent by this study.
The latest shared-key usage change is US$109.25905; its difference from our accounted total is US$0.00000. At the final check the difference is zero: the shared-key usage increase matches our conservative accounting exactly. The key is shared and provider totals can update at different times. This is an aggregate cross-check, not perfect request-level attribution. The earlier uncertain calls lack generation IDs, so their exact individual charges cannot be reconstructed from the saved record. New streamed calls preserve those IDs. See the final reconciliation and original reconciliation, current accounting snapshot and all local call receipts.
OpenRouter documents account, key and in-flight credit limits, as well as generation metadata lookup. Those explain the accounting mechanisms; the study's actual charges come from our saved receipts and account snapshots. Sources: credit limits, generation metadata.
Start with the task that resembles your work. Inspect the original file, the issue the reviewer identified, and the revision. A model that produces correct prose but puts zero into an unknown monetary field creates a different risk from one that misses a paragraph in a handoff. The usability threshold is designed to expose those weak criteria rather than hide them in an average.
The source-packet tasks test reconciliation without the uncertainty of live search. The laptop exercise therefore says nothing about current prices or buying recommendations. Clean invoice images do not establish OCR reliability on damaged scans. A roughly 6,000-word archive does not establish million-token performance. Working local code does not establish production security or maintainability. Two repetitions are too few to estimate rare failures.
A self-review can help, but it is still the same model inspecting its own work without external corrections. The revision columns show what happened here; they do not justify assuming that every error will be caught. Where the first attempt failed to save a deliverable, that failure remains part of the buying decision even if revision recovered it.
The bundle includes all 200 planned rows, exact prompts and fixture hashes, initial and revised files, visible answers and tool traces, grading notes, operator checks, earlier attempts, runner source and cost receipts. Shareable traces omit credentials, private account metadata and private reasoning fields. The full private response archive is retained separately. This is a sanitized evidence release, not an “unredacted” transcript.
Every prompt, input, output, transcript, grade and receipt is linked from the evidence index. The complete download preserves the folder structure for offline inspection. Privacy redactions are documented with original and public file hashes; they remove local user paths and private account-wide metadata, not substantive answers or scores.
Related Kingy testing: Qwen4-Max versus Sol, Qwen4-Max versus Opus. These are separate studies with different tasks and settings; their scores are not pooled with this one.