AI News

Qwen4-Max vs Astra, Sol, Opus and Fable: 200 task stages

Study status: 200/200 stages have reached a reported outcome; 193 returned a final response and 200 have a source-based grade. The full matrix is evaluated.

We gave five models the same work: reconcile a messy month of sales, choose a laptop, repair a small website, recover project decisions, write from a source packet, diagnose an export problem, triage customer tickets, schedule a meeting, check invoice images and extract current requirements from a long archive. Each task was run twice. Every initial submission then received the same self-review request, without the answer key.

The question was practical: what can someone use, what needs fixing, and what did it cost to get there? This is an inspection of returned work, not a claim that a model can run a company. Every business, customer, offer, invoice and software screenshot in the test is a constructed fixture. The tasks resemble real work while giving us known facts to check.

Open all 200 result rows and their files. Download the exact candidate inputs. Read the grading rubric. The downloadable guide-and-evidence bundle keeps these links together: extract it before opening the guide or evidence index.

The comparison at a glance

Initial submissions meeting the rubric: Astra20 of20, Sol19, Qwen18, Opus18 and Fable16; two repetitions of ten tasks.

A result meets our usability threshold only if it scores at least 16/20, scores at least 3/4 on each criterion, and has no defined critical failure. “Minor edits” meets that threshold; “major repair” does not. Five strong totals cannot compensate for one missing essential deliverable.

Model Initial usable / graded Initial mean /20 Revised usable / graded Revised mean /20 Median initial minutes Receipted study cost, USD
Qwen4-Max 18/20 18.65 18/20 18.80 2.69 3.6174
GPT-6 Astra 20/20 19.80 20/20 19.85 1.28 17.4133
GPT-6.1 Sol 19/20 19.60 20/20 19.90 1.36 2.8383
Claude Opus 5.5 18/20 19.20 18/20 19.45 1.47 23.9202
Claude Fable 5.2 16/20 17.30 17/20 18.50 2.07 61.3015

What the paired revisions changed

Qwen4-Max: among 20 graded pairs, scores rose in 4, stayed equal in 15 and fell in 1. 0 crossed into the usable range; 0 crossed out.

GPT-6 Astra: among 20 graded pairs, scores rose in 1, stayed equal in 19 and fell in 0. 0 crossed into the usable range; 0 crossed out.

GPT-6.1 Sol: among 20 graded pairs, scores rose in 4, stayed equal in 16 and fell in 0. 1 crossed into the usable range; 0 crossed out.

Claude Opus 5.5: among 20 graded pairs, scores rose in 6, stayed equal in 13 and fell in 1. 0 crossed into the usable range; 0 crossed out.

Claude Fable 5.2: among 20 graded pairs, scores rose in 5, stayed equal in 15 and fell in 0. 1 crossed into the usable range; 0 crossed out.

What we would take from these results

Astra cleared our usability threshold on all 20 initial submissions and all 20 revisions. Sol cleared it on 19 of 20 initials and all 20 revisions; its missing initial website handoff was the exception. Those are the strongest observed delivery records in this particular kit. They are not proof of superiority on unseen tasks, and the grading was not blinded.

Qwen cleared the threshold on 18 of 20 initial submissions and 18 of 20 revisions. Its receipted cost is shown above and is worth comparing with the specific repairs needed, especially the structured refund fields. Opus also cleared 18 of 20 in both stages; its support outputs show why a high average can coexist with a material weak criterion. Fable cleared 16 of 20 initials and 17 of 20 revisions, with three no-file truncations across the second accounting and coordination attempts. Its other completed outputs should still be judged on their own evidence.

These counts include genuine limit-hit outcomes graded as delivered. They do not turn historical provider interruptions into model failures. We would use this study to shortlist models for a particular workflow, then test fresh examples from that workflow before relying on it.

Model-by-model reading

Qwen4-Max: 2 of 20 reviewed initial submissions earned full marks. Initial outputs below our usability threshold: task 7, repeat 1 (14/20), task 7, repeat 2 (17/20). The task tables below link the exact outputs and the separate revision outcomes.

GPT-6 Astra: 16 of 20 reviewed initial submissions earned full marks. Every reviewed initial output met the usability threshold. The task tables below link the exact outputs and the separate revision outcomes.

GPT-6.1 Sol: 14 of 20 reviewed initial submissions earned full marks. Initial outputs below our usability threshold: task 3, repeat 1 (17/20). The task tables below link the exact outputs and the separate revision outcomes.

Claude Opus 5.5: 10 of 20 reviewed initial submissions earned full marks. Initial outputs below our usability threshold: task 7, repeat 1 (16/20), task 7, repeat 2 (16/20). The task tables below link the exact outputs and the separate revision outcomes.

Claude Fable 5.2: 10 of 20 reviewed initial submissions earned full marks. Initial outputs below our usability threshold: task 1, repeat 2 (0/20), task 4, repeat 2 (0/20), task 7, repeat 1 (17/20), task 7, repeat 2 (16/20). The task tables below link the exact outputs and the separate revision outcomes.

The mistakes that mattered

The support queue is a useful caution about structured output. Several submissions entered numeric zero for an undecided refund while surrounding prose said the decision was unresolved. An imported CSV may lose those qualifications. The source record required unknown amounts to stay unknown; this is a material data-handling error, not a tone preference. The task-seven tables and source-linked reviews show exactly which models and repetitions did this.

The website task exposed a different weakness: spending the call budget on repeated checks can leave the handoff unfinished even when the app works. Conversely, reaching a turn limit does not prove the saved work is unusable. We inspect both the delivered files and the actual endpoint behavior. The initial and revision rows deliberately preserve these distinctions.

In the source-synthesis tasks, correct headline answers sometimes came with invented explanations: a conflict attributed to the wrong document, an owner assignment treated as settled, or an approval condition added to the brief. Small-sounding additions can change what a person does next. The reviews identify the unsupported passage and the correction rather than treating fluent language as evidence.

Fable's second accounting initial run exhausted the output allowance without saving the required deliverables. Its revision recovered them. On the second coordination run, both the initial and revision exhausted the allowance with no required files. These are observed delivery failures under the tested configuration, not proof that a different token budget or app would behave the same way. The accounting recovery and the coordination failure both remain visible.

Read each numerator together with its denominator. These averages describe ten chosen fixtures with two repetitions, not a statistically representative population of jobs. A one-point editorial difference is less informative than the linked error. Costs include locally receipted earlier attempts; the unresolved historical aggregate difference cannot honestly be assigned to a model. Timings span recovery waves and changing load, so they are observations rather than a controlled speed ranking.

What we actually tested

There were 100 initial runs: ten tasks × five models × two fresh repetitions. Each had one fixed revision stage, producing 200 planned outcomes. The second repetition uses the same fixture; it checks repeatability on that fixture, not generalization to unseen examples. A revision may recover a failed delivery, but its score never replaces the first score.

Family Exact requested model Pinned provider Reasoning setting
Qwen 4 qwen/qwen4-max Alibaba high
Astra openai/gpt-6-astra OpenAI medium
Sol openai/gpt-6.1-sol OpenAI medium
Opus anthropic/claude-opus-5.5 Anthropic high
Fable anthropic/claude-fable-5.2 Anthropic high

These are OpenRouter API tests with provider fallback disabled, not tests of the vendors' chat apps. We explicitly selected the reasoning defaults observed in the catalog. Equal prompts and tools do not make these equal-compute configurations. Saved catalog and endpoint snapshots establish the identifiers and pricing available at execution.

Every candidate received the shared instructions and only its active task. Available tools could list, read and write local files, execute standard-library Python and inspect a local browser. Candidate tools had no external network access and could not read credentials, grading keys or other runs. Candidate Python also could not bind a local socket; the browser tool supplied the documented local API instead. Models attempting to launch their own server received a permission error and had to use the available tools. We preserve those errors because this environment constraint influenced the work. Text and CSV contents were supplied directly; image tasks received the same PNG inputs. The browser returned text, controls and overflow measurements, not screenshots.

Each stage had up to 16 model calls, with 16,384 output tokens per call including reasoning, and a nominal 30-minute initial or 15-minute revision allowance. A request already in flight could finish after that time. Early tasks had a 220,000-byte request cap; later tasks and recovery runs used 800,000 bytes. Initial batches ran three jobs concurrently; recovery used two and streamed responses to preserve generation IDs early. These changes limit timing comparisons. The exact history is in method corrections and the execution settings.

Infrastructure interruptions were retried in separately identified recovery attempts. Missing initial work restarted from the task inputs. Missing revisions resumed the saved initial conversation and restored the initial output folder; incidental workspace scratch files could remain. This is a disclosed recovery limitation. Genuine turn limits and output truncation were retained as outcomes rather than retried for a better score.

How the outputs were checked

Each task has five criteria worth 0–4 points. The answer keys existed outside candidate workspaces. A separate AI reviewer checked output against the source packet and recorded criterion-specific evidence, issues and file hashes. The reviewer could see model identities: this was unblinded AI review, not a blinded human panel. The operator also performed deterministic checks and actual website integration tests. Claims about correctness rest on source reconciliation and those checks, not on model agreement alone.

For the website, the candidate browser tool served a trusted implementation of the supplied lead API. That let models inspect frontend behavior without exposing an arbitrary server. Separately, the operator ran each delivered Python server in a disposable copy, checked persistence and invalid requests, verified prices, and captured 390- and 1,280-pixel views. A page can work while its required handoff is still missing; those receive separate credit.

We did not measure human checking or repair time. A model's estimate of time saved is not a measurement and is excluded. “Complete” in an API log means the response ended; it does not mean the work was correct. The grading sheets preserve that distinction.

Task-by-task results

Tables show both repetitions and both stages. Each cell gives the grade, usability category and execution status where it was not a normal completion. The time and cost columns show initial → revision for the selected attempts. All earlier attempts and their charges are linked under Files.

1. Messy business accounts

Assignment: Reconcile sales, refunds and advertising.

The checkable result is nine orders, CAD 2,820 gross sales, CAD 300 in applied refunds and CAD 450 in advertising. Net revenue of CAD 2,520 and contribution after ads of CAD 2,070 remain provisional because a settled Course refund has no amount. Workshop totals are unaffected. Another refund has no matching order and must remain unallocated. Contribution after ads is not business profit.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 18/20; minor edits 18/20; minor edits 3.66 → 2.40 0.12818 → 0.09011 Files
Qwen4-Max 2 19/20; minor edits 19/20; minor edits 3.63 → 3.32 0.13885 → 0.11890 Files
GPT-6 Astra 1 20/20; usable 20/20; usable 1.70 → 1.96 0.40114 → 0.53620 Files
GPT-6 Astra 2 20/20; usable 20/20; usable 1.71 → 1.46 0.40243 → 0.67919 Files
GPT-6.1 Sol 1 20/20; usable 20/20; usable 1.45 → 1.53 0.07905 → 0.09904 Files
GPT-6.1 Sol 2 20/20; usable 20/20; usable 1.93 → 0.88 0.07704 → 0.09244 Files
Claude Opus 5.5 1 19/20; minor edits 20/20; usable 1.61 → 1.69 0.61298 → 0.80026 Files
Claude Opus 5.5 2 20/20; usable 19/20; minor edits 1.75 → 1.26 0.41643 → 0.92980 Files
Claude Fable 5.2 1 19/20; minor edits 20/20; usable 3.38 → 2.13 2.20970 → 2.34745 Files
Claude Fable 5.2 2 0/20; failed; output_truncated 20/20; usable 2.33 → 2.86 0.68170 → 2.63500 Files

What to inspect: Qwen4-Max, repeat 1: RF04 is currently excluded but still requires reconciliation and could change final results. (output) Claude Opus 5.5, repeat 1: Refund on an unknown order; N9999 is absent. (output) Claude Fable 5.2, repeat 1: Only ad spend is assigned to products; sales-to-ad attribution is not given. (output)

2. A purchase decision

Assignment: Choose a laptop within a landed-cost and delivery budget.

Only offer O2 meets the stated requirements under the scenario GPU interpretation. Its landed price is CAD 2,098.88. The challenge is reconciling price, specifications, tax, shipping and delivery evidence; the offers are fictional and this is not a live buying recommendation.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 19/20; minor edits 19/20; minor edits 1.47 → 2.24 0.03793 → 0.07512 Files
Qwen4-Max 2 19/20; minor edits 20/20; usable 2.35 → 1.57 0.07988 → 0.06871 Files
GPT-6 Astra 1 20/20; usable 20/20; usable 0.81 → 1.22 0.21863 → 0.35603 Files
GPT-6 Astra 2 20/20; usable 20/20; usable 0.93 → 1.00 0.22004 → 0.32120 Files
GPT-6.1 Sol 1 20/20; usable 20/20; usable 0.76 → 0.93 0.05031 → 0.06249 Files
GPT-6.1 Sol 2 20/20; usable 20/20; usable 1.77 → 2.11 0.04680 → 0.06424 Files
Claude Opus 5.5 1 20/20; usable 20/20; usable 1.07 → 1.11 0.27598 → 0.47794 Files
Claude Opus 5.5 2 20/20; usable 20/20; usable 1.19 → 1.58 0.25583 → 0.85035 Files
Claude Fable 5.2 1 20/20; usable 20/20; usable 1.41 → 2.41 0.65512 → 1.74429 Files
Claude Fable 5.2 2 20/20; usable 20/20; usable 1.33 → 1.85 0.60769 → 1.60013 Files

What to inspect: Qwen4-Max, repeat 1: O5 has a differently named GPU and no supplied performance test; omit comparative strength unless source verifies it. (output) Claude Opus 5.5, repeat 1: Tied at 2.3kg with the other Forge SKU; P16-32 is heavier at 2.5kg. Does not affect purchase decision. (output)

3. A broken website

Assignment: Repair signup persistence, pricing and mobile layout.

The page posts to the wrong endpoint, applies the discount incorrectly and overflows on mobile. The checks run the delivered server and verify real saved records, invalid requests, four totals and two viewport widths. A working site and a complete handoff are scored separately.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 19/20; minor edits 19/20; minor edits 4.55 → 4.07 0.12196 → 0.12975 Files
Qwen4-Max 2 20/20; usable; turn_limit 20/20; usable 4.27 → 3.28 0.13952 → 0.14781 Files
GPT-6 Astra 1 20/20; usable 20/20; usable; turn_limit 3.16 → 2.00 0.81461 → 1.14669 Files
GPT-6 Astra 2 20/20; usable 20/20; usable 2.29 → 2.50 0.64110 → 0.90385 Files
GPT-6.1 Sol 1 17/20; major repair; turn_limit 20/20; usable 1.91 → 1.95 0.08201 → 0.08290 Files
GPT-6.1 Sol 2 20/20; usable; turn_limit 20/20; usable 2.12 → 1.55 0.12813 → 0.12517 Files
Claude Opus 5.5 1 20/20; usable 20/20; usable 2.66 → 2.66 0.95239 → 2.11108 Files
Claude Opus 5.5 2 20/20; usable 20/20; usable 2.53 → 1.86 1.13494 → 1.40546 Files
Claude Fable 5.2 1 20/20; usable 20/20; usable 4.02 → 3.12 2.56330 → 2.67105 Files
Claude Fable 5.2 2 20/20; usable 20/20; usable 4.16 → 4.10 3.14998 → 4.63975 Files

What to inspect: Qwen4-Max, repeat 1: Original supplied blank/0/fractional/out-of-range numeric inputs produce wrong numeric totals; NaN case not evidenced. (output) GPT-6.1 Sol, repeat 1: Supply handoff/run instructions/check evidence and remove fake submissions from runnable package. (output) Claude Opus 5.5, repeat 2: toggleAttribute aria-invalid yields empty attribute rather than literal true. Does not change required validation behavior, but use setAttribute true/false for accessibility cleanup. (output)

4. Project coordination

Assignment: Resolve old decisions and a later approved change.

The approved event is October 30 at 11:00 Pacific, for 80 people, free, on MeetRoom. Recording needs consent and paid ads remain on hold. Unassigned testing and recipient permission must remain open rather than be invented.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 18/20; minor edits 19/20; minor edits 2.67 → 2.34 0.07216 → 0.11110 Files
Qwen4-Max 2 19/20; minor edits 19/20; minor edits 2.00 → 2.69 0.06173 → 0.12070 Files
GPT-6 Astra 1 20/20; usable 20/20; usable 2.25 → 2.02 0.38701 → 0.46338 Files
GPT-6 Astra 2 19/20; minor edits 19/20; minor edits 1.86 → 1.59 0.38488 → 0.57925 Files
GPT-6.1 Sol 1 19/20; minor edits 20/20; usable 2.06 → 2.09 0.05514 → 0.09999 Files
GPT-6.1 Sol 2 19/20; minor edits 19/20; minor edits 1.46 → 1.27 0.06541 → 0.09074 Files
Claude Opus 5.5 1 19/20; minor edits 19/20; minor edits 1.76 → 2.11 0.41660 → 1.22648 Files
Claude Opus 5.5 2 19/20; minor edits 19/20; minor edits 2.05 → 1.37 0.80459 → 0.67347 Files
Claude Fable 5.2 1 19/20; minor edits 20/20; usable 3.26 → 3.29 1.28666 → 1.59423 Files
Claude Fable 5.2 2 0/20; failed; output_truncated 0/20; failed; output_truncated 2.49 → 2.21 0.67831 → 1.01057 Files

What to inspect: Qwen4-Max, repeat 1: Campaign approver unspecified; proposed Priya if desired, not confirmed. (output) GPT-6 Astra, repeat 2: Recipient-scope decision is delegated to an unassigned verifier or decision-maker, although Priya is coordinator and Mei explicitly asks her to resolve it. Assign Priya accountable resolution; she may delegate verification. (output) GPT-6.1 Sol, repeat 1: Approver unspecified; Priya may be proposed coordinator but authority not established. (output)

5. An editorial assignment

Assignment: Write accurately from a fictional AI-release packet.

The supported price is USD 29 with three seats minimum. An erroneous launch screenshot must not override the corrected source. Vendor draft-time savings are not end-to-end savings. The article must not invent hands-on testing, customer quotes, privacy guarantees or certification.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 19/20; minor edits 19/20; minor edits 2.71 → 2.32 0.06942 → 0.07074 Files
Qwen4-Max 2 19/20; minor edits 19/20; minor edits 1.89 → 2.28 0.07234 → 0.08200 Files
GPT-6 Astra 1 20/20; usable 20/20; usable 1.51 → 1.95 0.29422 → 0.52611 Files
GPT-6 Astra 2 20/20; usable 20/20; usable 1.31 → 1.55 0.28194 → 0.49826 Files
GPT-6.1 Sol 1 20/20; usable 20/20; usable 1.39 → 1.61 0.04998 → 0.06945 Files
GPT-6.1 Sol 2 20/20; usable 20/20; usable 1.30 → 0.76 0.05280 → 0.06014 Files
Claude Opus 5.5 1 20/20; usable 20/20; usable 1.52 → 1.57 0.34373 → 0.56879 Files
Claude Opus 5.5 2 20/20; usable 20/20; usable 1.19 → 1.15 0.28662 → 0.62060 Files
Claude Fable 5.2 1 19/20; minor edits 19/20; minor edits 2.53 → 2.59 0.98804 → 1.74074 Files
Claude Fable 5.2 2 18/20; minor edits 19/20; minor edits 1.85 → 2.31 0.78039 → 1.99872 Files

What to inspect: Qwen4-Max, repeat 1: The source provides waitlist access but no fee terms; say newcomers can join waitlist. (output) Claude Fable 5.2, repeat 1: API/EUaccess is planned and not currently available under source packet. (output)

6. Visual software support

Assignment: Diagnose export failure from three screenshots.

The blocking problem is offline media. Relink the original clip, preserve the edit, switch captions from a sidecar file to burned-in if they must appear in the MP4, and review the nonblocking audio warning. The model must not claim it operated the app.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 20/20; usable 20/20; usable 1.16 → 1.73 0.02760 → 0.03384 Files
Qwen4-Max 2 19/20; minor edits 19/20; minor edits 1.00 → 1.78 0.02665 → 0.04533 Files
GPT-6 Astra 1 20/20; usable 20/20; usable 0.75 → 0.74 0.16037 → 0.14852 Files
GPT-6 Astra 2 19/20; minor edits 19/20; minor edits 0.53 → 0.67 0.15519 → 0.14549 Files
GPT-6.1 Sol 1 20/20; usable 20/20; usable 0.65 → 0.72 0.02895 → 0.02229 Files
GPT-6.1 Sol 2 19/20; minor edits 19/20; minor edits 0.43 → 0.76 0.02922 → 0.03155 Files
Claude Opus 5.5 1 20/20; usable 20/20; usable 1.10 → 0.92 0.22033 → 0.25100 Files
Claude Opus 5.5 2 19/20; minor edits 19/20; minor edits 0.76 → 1.27 0.21769 → 0.73012 Files
Claude Fable 5.2 1 20/20; usable 20/20; usable 1.36 → 2.08 0.55947 → 0.99602 Files
Claude Fable 5.2 2 19/20; minor edits 19/20; minor edits 1.11 → 2.06 0.60130 → 1.88536 Files

What to inspect: Qwen4-Max, repeat 2: Final verification should explicitly check the 8–24-second interview segment, subtitle timing/synchronization and audio around 42 seconds in exported MP4; current step 6 mentions only visible captions and interview plays. (output) GPT-6 Astra, repeat 2: Include explicit subtitle synchronization/timing check in resulting MP4; identify the 8–24-second review range clearly in final playback instruction. (output) GPT-6.1 Sol, repeat 2: Include explicit subtitle synchronization/timing check in resulting MP4; identify the 8–24-second review range clearly in final playback instruction. (output)

7. Customer support

Assignment: Triage ten tickets against a store policy.

The queue tests return-window boundaries, a 20% course-viewing threshold, pending versus settled payments, a dangerous adapter and missing identity. An unknown refund is not numeric zero. Draft replies must not imply an action has already been performed.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 14/20; major repair 15/20; major repair 3.53 → 2.42 0.10451 → 0.10489 Files
Qwen4-Max 2 17/20; major repair 17/20; major repair 3.94 → 1.86 0.11445 → 0.06115 Files
GPT-6 Astra 1 19/20; minor edits 20/20; usable 1.45 → 1.38 0.35221 → 0.46533 Files
GPT-6 Astra 2 20/20; usable 20/20; usable 1.26 → 1.30 0.26660 → 0.36601 Files
GPT-6.1 Sol 1 20/20; usable 20/20; usable 1.18 → 0.85 0.05454 → 0.05393 Files
GPT-6.1 Sol 2 20/20; usable 20/20; usable 1.29 → 1.14 0.05308 → 0.05586 Files
Claude Opus 5.5 1 16/20; major repair 17/20; major repair 1.05 → 0.71 0.21179 → 0.23762 Files
Claude Opus 5.5 2 16/20; major repair 17/20; major repair 1.79 → 1.95 0.29066 → 0.70858 Files
Claude Fable 5.2 1 17/20; major repair 17/20; major repair 2.07 → 1.94 0.83135 → 1.34273 Files
Claude Fable 5.2 2 16/20; major repair 16/20; major repair 2.06 → 1.72 0.82314 → 1.00605 Files

What to inspect: Qwen4-Max, repeat 1: Source A704is2026-10-09,day11. (output) GPT-6 Astra, repeat 1: Acknowledge refund request without invented motive. (output) Claude Opus 5.5, repeat 1: Use blank/null for undetermined compensation and missing-order amount. (output)

8. Scheduling

Assignment: Find the earliest feasible meeting under resource constraints.

The earliest review is 14:00–14:45 Pacific in Room C, with preparation at 13:30, setup at 13:50 and follow-up at 14:45. The security decision remains unresolved. Availability, setup and follow-up all constrain the answer.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 19/20; minor edits 19/20; minor edits 2.46 → 2.66 0.06911 → 0.08976 Files
Qwen4-Max 2 19/20; minor edits 19/20; minor edits 2.24 → 2.54 0.07523 → 0.08128 Files
GPT-6 Astra 1 20/20; usable 20/20; usable 1.17 → 1.16 0.25852 → 0.30926 Files
GPT-6 Astra 2 20/20; usable 20/20; usable 1.14 → 1.45 0.23179 → 0.32698 Files
GPT-6.1 Sol 1 20/20; usable 20/20; usable 0.99 → 0.92 0.04754 → 0.04634 Files
GPT-6.1 Sol 2 19/20; minor edits 20/20; usable 1.33 → 0.95 0.04744 → 0.03653 Files
Claude Opus 5.5 1 19/20; minor edits 20/20; usable 1.19 → 1.32 0.27925 → 0.45598 Files
Claude Opus 5.5 2 19/20; minor edits 19/20; minor edits 1.42 → 1.18 0.27214 → 0.29789 Files
Claude Fable 5.2 1 20/20; usable 20/20; usable 2.10 → 2.01 0.86816 → 1.60974 Files
Claude Fable 5.2 2 20/20; usable 20/20; usable 1.63 → 1.69 0.52714 → 1.06790 Files

What to inspect: Qwen4-Max, repeat 1: Room schedule is supplied; assistant availability is an assumption, not provided. (output) GPT-6.1 Sol, repeat 2: Separate identifying/requesting security approval from booking the otherwise feasible review. (output) Claude Opus 5.5, repeat 1: Keep ownership unresolved without asserting it blocks sending the scheduling invitation. (output)

9. Invoice images

Assignment: Extract and reconcile two invoices.

Invoice 001083 prints 329.20, but its lines and tax total 319.20. It bills three microphones against two authorized and changes the bank suffix from 8821 to 8891. Verification and a payment hold are appropriate; the evidence does not prove fraud.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 19/20; minor edits 19/20; minor edits 1.87 → 2.35 0.03853 → 0.04530 Files
Qwen4-Max 2 19/20; minor edits 19/20; minor edits 3.11 → 1.57 0.08034 → 0.05565 Files
GPT-6 Astra 1 20/20; usable 20/20; usable 1.05 → 0.95 0.24485 → 0.24019 Files
GPT-6 Astra 2 20/20; usable 20/20; usable 1.29 → 0.68 0.18129 → 0.29208 Files
GPT-6.1 Sol 1 20/20; usable 20/20; usable 1.58 → 1.12 0.05403 → 0.03235 Files
GPT-6.1 Sol 2 20/20; usable 20/20; usable 0.84 → 0.72 0.05246 → 0.03943 Files
Claude Opus 5.5 1 19/20; minor edits 20/20; usable 1.21 → 1.13 0.32314 → 0.38278 Files
Claude Opus 5.5 2 20/20; usable 20/20; usable 0.96 → 0.59 0.20404 → 0.20940 Files
Claude Fable 5.2 1 20/20; usable 20/20; usable 1.99 → 1.51 0.91500 → 0.90250 Files
Claude Fable 5.2 2 20/20; usable 20/20; usable 1.52 → 1.34 0.59564 → 0.78603 Files

What to inspect: Qwen4-Max, repeat 1: Footer states shipping tax exemption; it does not authenticate shipping approval. Mark as evidence gap. (output) Claude Opus 5.5, repeat 1: Describe mismatch and trusted-contact verification; do not imply a supplied APprocedure or sourced prevalence claim. (output)

10. A long project archive

Assignment: Recover current scope and launch prerequisites.

Current scope excludes email alerts and SSO, limits the pilot to 80 accounts and requires an accessibility fix, rollback plan, written signoff and trained operator. A reserved deployment slot is not launch authorization. Instructions quoted from a customer remain source data rather than authority over the task.

Model Repeat Initial Revision Minutes, initial → revised USD, initial → revised Evidence
Qwen4-Max 1 19/20; minor edits 18/20; minor edits 2.72 → 2.05 0.07273 → 0.07151 Files
Qwen4-Max 2 19/20; minor edits 20/20; usable 3.22 → 2.22 0.07689 → 0.09260 Files
GPT-6 Astra 1 20/20; usable 20/20; usable 1.09 → 1.00 0.46915 → 0.64626 Files
GPT-6 Astra 2 19/20; minor edits 19/20; minor edits 1.07 → 1.19 0.47647 → 0.51443 Files
GPT-6.1 Sol 1 20/20; usable 20/20; usable 2.20 → 2.30 0.04378 → 0.08177 Files
GPT-6.1 Sol 2 19/20; minor edits 20/20; usable 1.08 → 1.16 0.08448 → 0.07813 Files
Claude Opus 5.5 1 20/20; usable 20/20; usable 2.13 → 2.15 0.67461 → 0.98743 Files
Claude Opus 5.5 2 19/20; minor edits 20/20; usable 1.87 → 1.76 0.42832 → 0.44180 Files
Claude Fable 5.2 1 19/20; minor edits 20/20; usable 2.27 → 2.66 1.52367 → 1.98448 Files
Claude Fable 5.2 2 20/20; usable 20/20; usable 1.68 → 2.26 0.92692 → 1.63811 Files

What to inspect: Qwen4-Max, repeat 1: Smoke executor, pilot-opening owner and reserved-slot owner unspecified; label inferred roles. (output) GPT-6 Astra, repeat 2: may deploy, no earlier than the conditional October 30 target (output) GPT-6.1 Sol, repeat 2: Asserts deployment cannot occur before October 30; source defines earliest pilot target, not separate prelaunch deployment prohibition. Keep target conditional without invented deployment-date limit. (output)

Costs, interruptions and the credit purchase

At the latest saved accounting check, cumulative inference is conservatively accounted at US$109.25905 against the authorized US$125 ceiling. This comprises US$67.5263644 in reconciled historical usage plus US$41.73269 in completion-pass receipts. Pending new-call reservations are US$0.00000. A reservation is a maximum allowance, not a confirmed charge.

The historical response receipts totaled US$67.3580344. The key's aggregate usage change was US$67.5263644, leaving US$0.16833 unassigned. That difference stays conservatively in the study accounting. Earlier unknown-call reservations totaled US$26.70488375; adding those to reconciled usage would double-count uncertainty as if it were confirmed spend. We preserve the original ledgers and distinguish them from reconciliation.

The approved top-up purchased US$50 in credits for a US$59.08 cash payment: US$50 credit principal, US$2.75 fee and US$6.33 tax. The credit purchase and later inference consumption are two views of the same funding, so they are not added together as model-test cost. Fees and tax are separately disclosed cash costs. Remaining credits are available account funds, not evidence that they were spent by this study.

The latest shared-key usage change is US$109.25905; its difference from our accounted total is US$0.00000. At the final check the difference is zero: the shared-key usage increase matches our conservative accounting exactly. The key is shared and provider totals can update at different times. This is an aggregate cross-check, not perfect request-level attribution. The earlier uncertain calls lack generation IDs, so their exact individual charges cannot be reconstructed from the saved record. New streamed calls preserve those IDs. See the final reconciliation and original reconciliation, current accounting snapshot and all local call receipts.

OpenRouter documents account, key and in-flight credit limits, as well as generation metadata lookup. Those explain the accounting mechanisms; the study's actual charges come from our saved receipts and account snapshots. Sources: credit limits, generation metadata.

Receipt2026-11-15
Tested
Ten synthetic practical tasks × five models × two repetitions, each with one fixed self-review: 200 graded stages.
Machine
Shared macOS workstation; isolated task workspaces. Model inference ran on provider infrastructure.
Runner
Kingy Python local-tools runner via OpenRouter; source, prompts and request receipts publicly linked.
Cloud side
Qwen4-Max/Alibaba; GPT-6 Astra and GPT-6.1 Sol/OpenAI; Claude Opus5.5 and Fable5.2/Anthropic. Provider fallback disabled.
Settings
16 calls/stage; 16,384 output tokens/call including reasoning. High for Qwen/Opus/Fable; medium for Astra/Sol. Recovery protocol changes disclosed.
Grading
Five criteria, 0–4 each. Usable requires ≥16/20, every criterion ≥3 and no critical failure. Unblinded AI review plus operator checks; 200 grades.
Cost
US$109.2590515 accounted inference, including US$0.16833 unassigned historical usage; no calls pending. US$59.08 bought US$50 credits; not double-counted.
Outcomes
193 final responses, four turn-limit outcomes and three output truncations; all graded as delivered.
Limitations
Synthetic fixtures, two repetitions, unblinded grading, differing reasoning defaults and recovery waves; not a general intelligence ranking.
Evidence privacy
Public copies omit private reasoning, credentials, account-wide metadata and local user paths. Original/public hash mappings disclose redactions.
Dates
run 2026-11-15 · checked 2026-11-16

Limits and how to use this evidence

Start with the task that resembles your work. Inspect the original file, the issue the reviewer identified, and the revision. A model that produces correct prose but puts zero into an unknown monetary field creates a different risk from one that misses a paragraph in a handoff. The usability threshold is designed to expose those weak criteria rather than hide them in an average.

The source-packet tasks test reconciliation without the uncertainty of live search. The laptop exercise therefore says nothing about current prices or buying recommendations. Clean invoice images do not establish OCR reliability on damaged scans. A roughly 6,000-word archive does not establish million-token performance. Working local code does not establish production security or maintainability. Two repetitions are too few to estimate rare failures.

A self-review can help, but it is still the same model inspecting its own work without external corrections. The revision columns show what happened here; they do not justify assuming that every error will be caught. Where the first attempt failed to save a deliverable, that failure remains part of the buying decision even if revision recovered it.

Transparency and the complete evidence archive

The bundle includes all 200 planned rows, exact prompts and fixture hashes, initial and revised files, visible answers and tool traces, grading notes, operator checks, earlier attempts, runner source and cost receipts. Shareable traces omit credentials, private account metadata and private reasoning fields. The full private response archive is retained separately. This is a sanitized evidence release, not an “unredacted” transcript.

Every prompt, input, output, transcript, grade and receipt is linked from the evidence index. The complete download preserves the folder structure for offline inspection. Privacy redactions are documented with original and public file hashes; they remove local user paths and private account-wide metadata, not substantive answers or scores.

Sources and downloads

Related Kingy testing: Qwen4-Max versus Sol, Qwen4-Max versus Opus. These are separate studies with different tasks and settings; their scores are not pooled with this one.