# Qwen4 Max versus GPT-6.1 Sol audit

Prepared November 15, 2026 Pacific. Frozen source capture: November 16, 2026 at 04:00:10 UTC. No new inference calls or credit consumption. Original study files were not changed.

## Decision

The five completed paired first-repetition tasks support a **small, explicitly bounded comparison**, not a general model ranking or the advertised ten-task, two-repetition study. Under the original rubric, both models passed four of five first attempts. Qwen cost $0.4617003 across those five first attempts; Sol cost $0.3134626. Sol's observed first-attempt spend was 32.1% lower. This includes unsuccessful attempts.

After the study's mandatory self-review pass on every task, Sol passed all five and Qwen passed four. Total spend for these ten stages per model was Qwen $0.9513309 and Sol $0.6581529. These are actual recorded workflow costs, not a simulation of stopping when a task first succeeds: the study also revised passing work.

All acceptance counts include the rubric's “minor edits” category. They do not mean error-free or ready to publish without review. Grading was by AI with visible model identities, not blinded human grading. The audit independently checked saved outputs against the synthetic source fixtures.

## First-attempt evidence

| Task | Qwen grade | Qwen USD | Sol grade | Sol USD |
| --- | --- | ---: | --- | ---: |
| T01 messy sales/refunds | 18/20, accepted with minor edits | 0.1281828 | 20/20, accepted | 0.0790497 |
| T02 fixed purchase-source packet | 19/20, accepted with minor edits | 0.0379323 | 20/20, accepted | 0.0503150 |
| T03 repair local website | 19/20, accepted with minor edits | 0.1219641 | 17/20, not accepted: handoff criterion 1/4, turn limit | 0.0820133 |
| T07 support triage | 14/20, not accepted | 0.1045068 | 20/20, accepted | 0.0545447 |
| T08 scheduling | 19/20, accepted with minor edits | 0.0691143 | 20/20, accepted | 0.0475399 |
| Total / accepted instances | 4/5 | **0.4617003** | 4/5 | **0.3134626** |

Qwen was cheaper on T02, so do not claim Sol was cheaper on every task. Sol's website was functional but its initial run hit the common 16-turn limit and omitted the required handoff. Do not describe it as unable to repair the app.

The original acceptance rule is total at least 16, every criterion at least 3, and no critical failure. A 17/20 score can fail; a 19/20 result can pass with minor edits. Do not infer acceptance from total score alone.

## After the mandatory self-review stage

| Task | Qwen final grade | Qwen initial + review USD | Sol final grade | Sol initial + review USD |
| --- | --- | ---: | --- | ---: |
| T01 | 18/20, minor edits remain | 0.2182926 | 20/20 | 0.1780860 |
| T02 | 19/20, minor edits remain | 0.1130523 | 20/20 | 0.1128058 |
| T03 | 19/20, minor edits remain | 0.2517168 | 20/20 | 0.1649150 |
| T07 | 15/20, fails payment-data criterion | 0.2093982 | 20/20 | 0.1084699 |
| T08 | 19/20, minor edits remain | 0.1588710 | 20/20 | 0.0938762 |
| Total / accepted instances | 4/5 | **0.9513309** | 5/5 | **0.6581529** |

Observed cost per accepted task, including the money spent on failed tasks:

- Initial-only: Qwen $0.115425075; Sol $0.07836565. Both denominators are four.
- Mandatory-review workflow: Qwen $0.237832725; Sol $0.13163058. Denominators are four and five respectively.

Use about **11.5¢ vs 7.8¢** for initial-only cost per accepted task, or **23.8¢ vs 13.2¢** for the mandatory-review workflow. State which workflow accompanies the figure. These sub-dollar results come from small synthetic fixtures with caching; they are not a price quote for a production workload.

## Costs independently reconciled

The audit checked **193 saved Qwen/Sol response receipts** across completed and partial work. Every receipt has a unique response ID, a matching ledger amount, a matching provider/model, non-BYOK attribution, and a total that reconciles with disjoint ordinary-input, cache-read, cache-write and output buckets. Tolerance was $0.000000001. No mismatches were found. The observed recorded charges total $1.7176697 across this whole snapshot, including partial/unpaired work; this is not the cost of the five-task comparison.

Reconstructed full request bodies from saved conversation prefixes and the tool definitions match **193/193 stored request SHA-256 hashes**. This verifies the requested settings rather than relying only on a mutable script: Qwen high, Sol medium, max_tokens 16,384, same five tools, fixed provider selection, fallback disabled, usage enabled and plugins empty.

Response metadata confirms Alibaba for Qwen and OpenAI/default service tier for Sol. The cost arithmetic confirms Sol Standard ($2 ordinary input/$0.10 reads/$2.50 writes/$10 output per million) and Qwen ($1.50 ordinary input/$0.075 reads/$4.50 output). No Flex pricing appears in the receipts. These are OpenRouter-returned charges and upstream cost details; an independently downloaded billing statement or settled payment invoice was not available. Say **recorded API charges**, not “audited invoice paid.”

For the five first attempts, Qwen consumed 48,887 billed output tokens, including 15,371 reported reasoning tokens; Sol consumed 19,085, including 1,484 reasoning tokens. Qwen used about 2.56 times the total billed output tokens at these different effort settings. Reasoning is already inside output, never an extra charge to add again.

Caching differed in the measured route. First attempts: Qwen 296,704 cache-read tokens and no cache writes; Sol 165,151 cache-read and 42,343 cache-write tokens. This was not an uncached controlled comparison. At the same observed token counts but ordinary input prices throughout, the arithmetic would be Qwen $0.8845035 and Sol $0.606078. These are counterfactual calculations, not new runs; caching can also change system behavior and timing.

## What the grading actually found

- **Qwen sales memo:** correct principal calculations, but it called all other exclusions final even though an orphan refund still required reconciliation. The uncertainty disclosure needs editing. Both original and reviewed versions retained this issue.
- **Qwen purchase memo:** correctly chose the qualifying offer, but called another laptop the stronger-GPU option without a supporting performance comparison in the supplied packet. Remove that claim. This was fixed-source synthesis, not live shopping or web research.
- **Sol website:** the first saved app passes functionality, but the initial run is incomplete because it lacks the required handoff. The later review supplies it. A turn-limit outcome is not a provider outage.
- **Qwen website handoff:** the working app passes, but its description of the original defect includes an unsupported NaN claim. The original browser-check count is also inconsistent. Verify against the saved trace rather than repeating the handoff's count.
- **Qwen support triage:** a purchase date is wrong; two unassessed refund amounts are written as zero; and the output says order/contact were confirmed privately when that had not happened. The revision did not resolve the material refund-data issue. Do not describe the whole task as successful.
- **Qwen scheduling memo:** its schedule is feasible in the fixture, but it incorrectly implies assistant availability was supplied in the calendars. It was an assumption. Label it as such.
- **Qwen invoice extraction, unpaired:** the independent reviewer checked the two invoice images and awarded 19/20. Extraction and the disputed-invoice arithmetic were correct, but the note treated an unverified CAD20 shipping charge on the other invoice as nonblocking. This can illustrate a review issue; it cannot establish a Qwen-versus-Sol result because the Sol counterpart was incomplete at the cutoff.

The independent audit retains the original pass/fail rule, with explicit errors. A stricter “no unsupported factual statements” policy would fail several Qwen memos; do not silently apply that new threshold and present it as the study's original scoring.

## Website verification

The audit replayed the four saved T03 applications locally, in disposable directories, with external browser requests blocked. It used the exact unmodified original Python server, which all four deliverables retained byte for byte. Both initial and revised versions for both models passed all 19 checks: prices, invalid seats, persisted valid submission, rejected invalid requests, and no horizontal overflow at 390 and 1280 pixels. Mobile screenshots were also inspected for basic readability. This is functional verification of the saved outputs, not new model execution or production-readiness certification.

The original candidate browser used a trusted server rather than the model's delivered server; this audit closes that specific gap for these four artifacts. It does not prove all handoff claims or general accessibility. Preserve Sol's missing initial handoff and Qwen's handoff inaccuracies.

## Incomplete population and boundaries

Snapshot contains five complete paired first-repetition tasks (T01,T02,T03,T07,T08), plus Qwen T09 initial without a complete Sol counterpart. T04 Qwen and T09 Qwen revision/Sol initial were still partial. Three unresolved reservations total $1.31214975. They are not confirmed charges and are excluded from paired totals. There are no completed second-repetition Qwen/Sol runs in this snapshot. Planned but absent tasks are not zero scores.

The source study continued changing during this audit; the snapshot is the cutoff. Later results belong in a separately captured extension, not silently added to this report. Costs from partial/unpaired work remain in the audit ledger, not the paired comparison denominator. Independent grading covered 21 completed output stages out of 80 planned Qwen/Sol stages: 20 paired stages and the unpaired Qwen invoice initial. The other 59 planned stages remain incomplete or absent. Snapshot integrity verified all 1,033 captured files without a hash mismatch.

The exact initial system/user payloads matched for all six pairs where both conversations exist. The core tools were list/read/write files, local Python and a text-returning browser. No live web retrieval, native product integrations or model-to-model delegation was provided. Candidate Python had an isolation profile; no answer-key access appeared in saved tool outputs. This is not a formal security proof.

## Recommended article use

Suggested title: **Qwen 4 Max vs GPT-6.1 Sol: Five Work Tasks, Failures and Recorded Costs**.

Safe opening: “On five small synthetic work tasks, Qwen4-Max and GPT-6.1 Sol each met our original acceptance rubric on four first attempts. Sol's recorded first-attempt API charges were $0.3135, versus Qwen's $0.4617. After a mandatory self-review stage, Sol met the rubric on all five; Qwen met it on four. These results compare Qwen high with Sol medium through OpenRouter, with different observed caching and one repetition.”

Use the counterexample: Qwen's fixed-source purchase memo was cheaper, while Sol's website run needed a second stage to finish the handoff. Show support-triage errors beside the actual source rows, not just a composite score. Round display figures, retain exact receipts in the evidence.

Do not claim: general superiority; statistical significance; ten completed tasks; two repetitions; error-free passing work; blinded human review; measured human repair time; a native ChatGPT-versus-Qwen app test; equal compute; an uncached test; or a complete paid invoice reconciliation. Do not label mandatory polishing spend as necessary repair cost.

The article can use these bounded findings now. It should keep results tables and all failures visible, identify synthetic data, link the prompts and rubric, and redact private response reasoning and account metadata before any public evidence release. No public evidence bundle or article was published by this audit.

## Audit artifacts

- `cost-audit.json`: every stage, receipt reconciliation and pending reservation.
- `request-reconstruction.json`: hash verification of all 193 observed requests.
- `paired-costs.csv`: machine-readable paired stage costs and original grades.
- `grading-audit.json` / `grading-audit.md`: independent source/output review.
- `site-checks/`: local application-check logs and screenshots.
- `snapshot-20261116T040010Z/snapshot-manifest.json`: the immutable audit input inventory and hashes.

All raw audit files are local/private. Snapshot includes original response fields that must be sanitized before public sharing.
