Qwen4-Max and GPT-6.1 Sol each met our original acceptance rubric on four of five first attempts at synthetic work tasks. Sol’s recorded first-attempt API charges totaled $0.3134626; Qwen’s totaled $0.4617003. That made Sol 32.1% cheaper across this small set, including the failed attempts.
The tasks covered a sales memo, a purchase recommendation, a website repair, support triage and scheduling. After a mandatory self-review stage on every task, Sol met the rubric on all five and Qwen on four. The combined initial and review charges were $0.6581529 for Sol and $0.9513309 for Qwen.
There are important boundaries. This was one repetition of five synthetic tasks, run through OpenRouter with Qwen set to high reasoning and Sol to medium. Caching differed. AI graders could see model identities, and the rubric accepted some work needing minor edits. The figures describe these recorded runs, not a general ranking or a production price quote. The evidence package includes the task materials and results.
In the Vals Index v2.2 on November 15, 2026, Qwen scored 63.20% and Sol 63.94%, at reported costs per test of $9.33 and $3.61 respectively. Qwen’s cost was about 2.58 times Sol’s; the 0.74-percentage-point score gap does not establish a decisive quality winner. These are Vals’ evaluator-specific results, separate from Kingy’s runs.
Download the complete redacted evidence package, or browse all five tasks and every initial and review stage.
What the first attempts produced
| Synthetic task | Qwen result | Qwen charge | Sol result | Sol charge |
|---|---|---|---|---|
| Sales and refunds | 18/20, accepted with minor edits | $0.1281828 | 20/20, accepted | $0.0790497 |
| Fixed-source purchase packet | 19/20, accepted with minor edits | $0.0379323 | 20/20, accepted | $0.0503150 |
| Repair a local website | 19/20, accepted with minor edits | $0.1219641 | 17/20, not accepted | $0.0820133 |
| Support triage | 14/20, not accepted | $0.1045068 | 20/20, accepted | $0.0545447 |
| Scheduling | 19/20, accepted with minor edits | $0.0691143 | 20/20, accepted | $0.0475399 |
| Total | 4/5 accepted | $0.4617003 | 4/5 accepted | $0.3134626 |
See the paired costs and grades. Qwen’s initial purchase task cost 3.79 cents, compared with Sol’s 5.03 cents, a saving of 1.24 cents. Including mandatory review, the totals were effectively equal: 11.305 cents for Qwen and 11.281 cents for Sol. Qwen selected the qualifying offer, but its memo also called another laptop the stronger-GPU option without a performance comparison in the supplied packet. That unsupported claim remained after review. The task used fixed sources; it was not a live shopping test.
Acceptance required at least 16 out of 20, at least three points on each criterion and no critical failure. Thus a 17 could fail and a 19 could pass with minor edits. Passing did not mean error-free or ready to publish without review.
A working site and a missing handoff
The website repair task shows why a score alone can mislead. Sol’s first saved application worked, but its run reached the common 16-turn limit without delivering the required handoff. It scored 17/20 and failed the handoff criterion. Its review stage supplied the missing handoff and passed.
Qwen’s initial website task passed, but its handoff described an original NaN defect without support in the saved evidence, and its browser-check count was inconsistent. The working application and the explanation of the work need separate scrutiny.
For an independent check, the audit replayed all four saved site versions locally: each model’s initial and revised app. All passed 19 checks covering prices, invalid seats, persisted valid submissions, rejected invalid requests and horizontal overflow at two screen widths. The replay used the unmodified delivered Python server and blocked external browser requests. It supports the specific functionality checked, not a claim of production readiness or complete accessibility.
The support failure was more than a low score
Qwen’s initial support triage scored 14/20 and its revised version scored 15/20, still below acceptance. The audit found a wrong purchase date, two unassessed refund amounts entered as zero, and a statement that order and contact information had been confirmed privately when the saved evidence did not show that confirmation. The revision left the material refund-data problem and false confirmation in place. Compare the saved triage file and revised triage file with the supplied policy.
A zero in a machine-readable refund field can look like a decision that no refund is owed. The audit found that these amounts had not been assessed. That distinction matters to anyone handing a triage file to another person or system.
Other accepted Qwen work also needed attention. Its sales memo treated exclusions as final despite an orphan refund requiring reconciliation. Its scheduling memo produced a feasible schedule for the fixture but presented assistant availability as supplied in the calendars, when it was an assumption. Those errors remained within the original rubric’s minor-edit allowance; they should not be erased by the pass count.
Sol’s support output met all five grading criteria in both stages. It left unassessed refund amounts blank and made verification conditions explicit. That is the most useful contrast in this task: a downstream reader can distinguish a known zero from a decision still waiting on evidence. See Sol’s revised triage file.
What the review stage cost
| Task | Qwen after review | Qwen both stages | Sol after review | Sol both stages |
|---|---|---|---|---|
| Sales and refunds | 18/20, accepted | $0.2182926 | 20/20, accepted | $0.1780860 |
| Purchase packet | 19/20, accepted | $0.1130523 | 20/20, accepted | $0.1128058 |
| Website | 19/20, accepted | $0.2517168 | 20/20, accepted | $0.1649150 |
| Support triage | 15/20, not accepted | $0.2093982 | 20/20, accepted | $0.1084699 |
| Scheduling | 19/20, accepted | $0.1588710 | 20/20, accepted | $0.0938762 |
| Total | 4/5 accepted | $0.9513309 | 5/5 accepted | $0.6581529 |
The review was mandatory even when the first answer had already passed. These totals therefore describe a two-stage workflow, not the cost of stopping as soon as a task succeeds. They also exclude any human time spent checking or editing the work.
Dividing recorded spend by accepted tasks, while including spend on failures, gives about 11.5 cents per accepted Qwen task and 7.8 cents per accepted Sol task for first attempts. For the mandatory-review workflow, the figures are about 23.8 cents and 13.2 cents respectively. These ratios are descriptive arithmetic on four or five accepted tasks, not reliable forecasts.

How long the recorded stages took
The saved timings give these summed stage durations across the five tasks:
| Stages included | Qwen | Sol |
|---|---|---|
| First attempts | 15.67 minutes | 6.29 minutes |
| First attempts plus mandatory review | 29.47 minutes | 12.48 minutes |
These figures include API waits, tool execution and saves made inside the run loop. They exclude setup, output copying, the independent audit and human review.
Runs could overlap, so these sums do not measure how long the whole batch took. With different reasoning settings and caching, and one repetition per task, they describe the recorded runs rather than establish typical speed.
Why the charges differ
For the first attempts, Qwen used 48,887 billed output tokens and Sol used 19,085. Those totals include reported reasoning tokens; reasoning tokens are not an extra charge to add on top. The settings differed, so the observed token difference cannot tell us what would happen at equal effort.
The saved charges reconcile to these rates per million tokens. They describe the tested routes at the time of the runs, not a promise of current pricing:
| Recorded billing category | Qwen | Sol |
|---|---|---|
| Ordinary input | $1.50 | $2.00 |
| Cache read | $0.075 | $0.10 |
| Cache write | No billed writes in these runs | $2.50 |
| Total generated output, including reasoning | $4.50 | $10.00 |
Qwen’s lower output rate did not offset its greater billed output volume in the five first attempts. Its output-token count was about 2.56 times Sol’s. That is an observation about these tasks and settings, not proof that Qwen always uses more tokens. The cost audit preserves the token categories and receipt checks.
Caching also differed: Qwen recorded 296,704 cache-read tokens and no cache writes, while Sol recorded 165,151 cache-read and 42,343 cache-write tokens. The audit reconciled saved response receipts to the recorded ledger, but did not obtain an independent settled invoice. These are recorded API charges, not a verified paid invoice.
The test used the same initial task payloads and core tools for the paired runs: file operations, local Python and a text-returning browser. It did not give the candidates live web retrieval, native product integrations or model-to-model delegation. The purchase task, for example, measures work with its supplied packet rather than current market research.
Test receipt
| Item | Recorded method |
|---|---|
| Models and settings | qwen/qwen4-max, high; openai/gpt-6.1-sol, medium |
| Routes | OpenRouter to Alibaba or OpenAI respectively; provider fallback disabled |
| Shared limits | 16 model turns per stage; 16,384 generated tokens per response; 30-minute initial and 15-minute review stage budgets |
| Input and tools | Same initial payloads; synthetic files; local Python and text browser observations |
| Grading | Unblinded AI review against source fixtures; minor edits allowed under the original acceptance rule |
| Cost evidence | Saved provider-returned charges reconciled against token categories and ledgers |
| Date and cutoff | November 15, 2026 Pacific; frozen November 16 at 04:00:10 UTC |
The broader audit checked 193 response receipts, including partial and unpaired work outside the five-task table, without finding a cost mismatch. It also reconstructed all 193 request hashes to verify the recorded settings. The downloadable package separates the paired results from that wider audit scope. Read the grading audit and redaction disclosure.
- Tested
- Five paired synthetic work tasks; one repetition per model; initial and mandatory self-review stages.
- Runner
- Common local tool runner through OpenRouter; 193 saved request hashes verified in broader audit snapshot.
- Cloud side
- qwen/qwen4-max via Alibaba, high; openai/gpt-6.1-sol via OpenAI, medium.
- Settings
- Same tools; provider fallback off; 16 model turns per stage; 16,384 generated tokens per response; caching as observed.
- Grading
- Unblinded AI source/output review; >=16/20, all criteria >=3/4, no critical failure. Minor edits count as accepted.
- Cost
- Five tasks: initial Qwen $0.4617003, Sol $0.3134626; including mandatory review Qwen $0.9513309, Sol $0.6581529. Provider-returned charges.
- Cutoff
- Frozen 2026-11-16 04:00:10 UTC (November 15 Pacific). Later results excluded.
- First acceptance
- Qwen 4/5, Sol 4/5; after mandatory self-review Qwen 4/5, Sol 5/5.
- Limits
- One small synthetic repetition; different effort and caching; no measured human repair time; no settled billing invoice.
- Dates
- run 2026-11-15 · checked 2026-11-15
Limits
The frozen capture is from November 16, 2026 at 04:00:10 UTC, or November 15 Pacific. Five paired first repetitions were complete. The larger plan included ten tasks and two repetitions, but incomplete or absent work was not scored as zero and is not part of these totals. A separate unpaired Qwen invoice task cannot establish a comparison with Sol.
AI grading was unblinded. The independent audit checked saved outputs against the synthetic source fixtures and retained the original rubric, including its allowance for minor edits. Five tasks and one repetition cannot establish statistical significance, typical failure rates or performance across a broader workload. Different reasoning settings and cache behavior also prevent an equal-compute or uncached comparison.
When the cheaper rate saves money
For your own workflow, compare API charges plus the cost of human checking and correction, divided by the number of acceptable completed tasks. Estimate the human cost as manual minutes multiplied by an hourly rate and divided by 60. Include failed work in the cost total. This comparison excludes other costs, such as infrastructure and business losses. A lower token rate can save money if extra token use, checking and corrections do not erase the difference.
We did not measure human review time, so this experiment cannot supply that full comparison. Model wait time also cannot be counted automatically as paid human work. Try representative tasks, define what counts as acceptable before running them, and log the manual minutes needed to check and correct each result. Keep the source records alongside the outputs, especially for fields that trigger payments or other downstream actions.
Sources and evidence
The findings come from Kingy’s frozen cost audit, independent grading review, exact recorded costs and the complete evidence index. These contain synthetic inputs, outputs, visible tool traces and the redaction disclosure. For general launch specifications, see the Qwen4-Max guide.
