Qwen3.8-Max vs Kimi K3 vs DeepSeek V4 Flash: 45 Controlled Runs

Five original tasks, three fresh attempts per model, official providers pinned and fallbacks disabled. Under a shared high-reasoning, 8,000-token limit, Kimi K3 was the safest model at getting an answer over the line; the reason matters more than the headline score.

We sent Qwen3.8-Max, Kimi K3, and DeepSeek-V4-Flash-0731 through the same controlled comparison. In 22 of the 45 runs, the user-facing answer never arrived.

Every request returned HTTP 200 from the intended model and provider, and every one passed the test harness’s route and configuration checks. The 22 missing answers came from token-budget exhaustion: Qwen3.8-Max and DeepSeek-V4-Flash-0731 repeatedly spent almost the entire allowance on reasoning, reached the limit, and left the visible answer empty.

Kimi K3 handled the same envelope much better. It returned visible text in 14 of 15 runs and achieved a descriptive mean of 96.8 across the five task medians. Qwen earned six 100-point results across constraint reasoning and structured extraction, but produced visible text in only six of 15 runs. DeepSeek solved all three constraint puzzles perfectly, yet produced no visible answer in its other 12 attempts.

Kimi K3 was the most reliable one-shot choice under this exact OpenRouter configuration. This small test does not establish the most intelligent model. It exposed a sharp difference in how three hosted services managed a shared reasoning-and-answer budget. A larger cap, a lower or explicit reasoning budget, another endpoint, or a direct-provider API could change the order.

What we tested, and what the names mean

The evidence cutoff was August 3, 2026. That date matters because these products were moving quickly.

Alibaba had just released production Qwen3.8-Max under the direct API ID qwen3.8-max. Qwen said weights would follow the next week, so the tested production model was hosted-only at the cutoff; it should not be described as open-weight yet. QwenCloud lists a one-million-token context window and 131,072 maximum completion tokens (Qwen launch, QwenCloud model page).

Moonshot AI’s hosted Kimi K3 uses the API ID kimi-k3, with a 1,048,576-token context window and a default 131,072-token completion cap. Public weights were available under Moonshot’s custom Kimi K3 License, so the safer description is open-weight rather than unqualified open source (Kimi quickstart, official repository, license).

DeepSeek-V4-Flash-0731 was accessible as a live API, though DeepSeek’s July 31 changelog called it a public beta. The direct service lists a one-million-token context window and 384,000 maximum output. Its exact weights were available under MIT (DeepSeek changelog, pricing and model table, model card).

We tested those hosted services, not their weights in isolation.

Methodology: a small test with hard edges

The suite contained five original, closed-book, text-only tasks: code repair, constraint reasoning, structured extraction, evidence synthesis, and editorial writing. Each task used one fixed prompt. Each model received that prompt in three new stateless conversations, producing 45 scored calls. There were no repair turns, previous answers, model-specific hints, web access, or supplied tools.

The suite was frozen before the final 45-call run. The code-repair task had also been used during excluded configuration pilots, but no pilot response was scored.

All requests went through OpenRouter, with both order and only set to the official provider endpoint: Alibaba for Qwen, Moonshot AI for Kimi, and DeepSeek for DeepSeek-V4-Flash-0731. Provider fallback was off and parameter support was required. The harness rejected a response if the completion envelope or separately retrieved generation metadata named the wrong provider or route. This is stricter than selecting a generic model page, where a router may choose a cheaper third party (OpenRouter provider routing).

The exact routed identities were:

The run archived the endpoint snapshots because live router catalogs can change. Current route metadata is exposed by OpenRouter for Qwen, Kimi, and DeepSeek.

Every call requested OpenRouter’s normalized high reasoning setting. This made the submitted control consistent, but not the underlying computation. Kimi and DeepSeek document native low, high, and max settings; Qwen documents low, medium, and xhigh. A router can translate normalized settings, and providers can differ in hidden prompts, tokenization, accounting, and implementation. “High” here does not mean compute-matched (OpenRouter reasoning guide, Kimi reasoning guide, DeepSeek Chat Completions reference).

The test imposed no more than 8,000 generated tokens per response, including reasoning. The production request set max_tokens to 7,998 and independently rejected a response if either OpenRouter’s completion count or the provider-native completion count exceeded 8,000. OpenRouter reports reasoning tokens as a subset of completion tokens, not an additional quantity (OpenRouter usage accounting).

Why 7,998 instead of 8,000? In an excluded pilot, Qwen received an 8,000-token request and returned a reported completion count of 8,002: 8,000 reasoning tokens plus two other completion tokens. We reduced the request cap by two while retaining the independent 8,000-token post-check. A later DeepSeek pilot returned a zero-charge 404 because the account’s data-policy guard did not permit the exact pinned endpoint. The final run began only after the setting was corrected and the three providers passed serialized canaries, in the order DeepSeek, Kimi, then Qwen. The scheduler then opened a three-call-wide window with no more than one live request per provider.

No temperature or top-p override was submitted, leaving each endpoint’s supported default behavior in place. No tools were supplied.

How the answers were scored

Tasks 1–3 used deterministic validators.

The code repair had ten tests worth ten points each. Candidate Python ran in a macOS Seatbelt profile with process, file, network, time, CPU, and memory restrictions. Before model code was evaluated, the reference solution scored 100 and hostile read, write, network, and subprocess probes were blocked. That is a controlled sandbox preflight, not a claim of perfect containment against every possible exploit.

The constraint puzzle had one exhaustively verified solution. Its scorer checked schema, one-to-one assignments, all 14 clues, and the final schedule. The extraction task used a field-level validator that applied a fictional event manifest’s amendments chronologically, then checked schema, values, omissions, and calculated totals.

For evidence synthesis and editorial writing, the answer text was separated from the API envelope and stripped of model, provider, run order, latency, cost, and token metadata. Two separately prompted AI reviewers received independently randomized blind packets and scored each response against task-specific 100-point rubrics. A third blinded AI adjudicator was required if the first totals differed by more than 12 points, they disagreed over whether a score cap applied, or either flagged a possible invention. One editorial response triggered adjudication. There was no human judging panel.

All three attempts counted. The primary task summary is the median of the three scores, with the individual scores shown beside it. The equal-weight mean of five task medians is descriptive only; it is not a statistically estimated win rate or a universal leaderboard score.

Results at a glance

Kimi’s 96.8 compresses two important failures. It had one zero in extraction and another in evidence synthesis; its pooled mean across all 15 individual scores was 84.1. The run-level record is more useful than the headline aggregate.

Task 1: code repair

The prompt supplied a multiply broken Python 3.11 inventory reconciler. The repair had to handle UTC ordering, stable ties, one-shot iterables, duplicate IDs, validation, inventory invariants, deterministic output, immutability, and fresh result objects. The requested response format was exactly one Python code fence.

Kimi was the only model to hand executable code to the validator. It scored 90, 100, and 100. The 90-point version passed nine tests but raised TypeError where the contract required ValueError. That is the kind of small exception-contract miss that ordinary “looks correct” review can overlook and a real test suite catches immediately.

Qwen and DeepSeek each ended all three runs at the token limit with reasoning present and no visible code. Their zeros do not show that a completed repair would have failed. They show that, within this envelope, no repair reached the caller.

Kimi’s median wall time was 138.83 seconds for this task, compared with 195.31 seconds for Qwen and 86.87 for DeepSeek. Those latter times bought no user-facing answer, which is why raw latency without completion state is a poor operational metric.

Task 2: constraint reasoning

The second task assigned eight systems and eight leads to eight migration slots under 14 coupled clues, with raw JSON as the required output. All nine responses scored 100 and stopped normally.

This is the most useful control result in the suite. It shows that Qwen and DeepSeek were correctly routed and capable of exact work when their reasoning fit inside the allowance. DeepSeek’s three calls cost $0.0021948472 and had a 23.06-second median wall time. Qwen cost $0.025644 at 23.58 seconds; Kimi cost $0.0570552 at 32.91 seconds.

On a compact puzzle with a crisp target, DeepSeek paired correctness with an extraordinary price advantage. No further qualification is needed for these nine outputs: every schema and clue check passed.

Task 3: structured extraction

The extraction task supplied a fictional event manifest plus out-of-order amendment memos. Models had to apply the memos chronologically, normalize times and emails, update prices and tags, preserve cancellations, ignore a distractor, and calculate capacity and face value. Only raw JSON was accepted.

Qwen was perfectly consistent: 100, 100, 100. This was its clearest strength in the test. Its median wall time was 103.41 seconds, slower than Kimi’s 59.22 seconds, but all three outputs obeyed the interface and matched the expected result.

Kimi scored 100, 0, and 100. The failed response contained visible JSON, but wrapped it in a Markdown json fence. The parser stopped at the first character because the contract explicitly required raw JSON. That zero is an interface-compliance failure, not evidence that the underlying extraction was simply nonsense. For an automated production pipeline, however, the distinction offers little comfort: invalid wire format is invalid output.

DeepSeek used all 7,998 completion tokens as reasoning in each run and returned no visible JSON. It scored zero three times.

Task 4: evidence synthesis

For this task, the model wrote a 900–1,100-word memo to a fictional refrigerated-logistics COO. The source packet mixed measured outcomes with vendor-defined metrics, an incorrect six-week annualization of an eight-week pilot, a missing compliance cost, weak causal evidence, an outage, and contradictory survey framing. A strong memo had to choose a decision, repair the arithmetic, preserve uncertainty, address counterevidence, and set actionable gates.

Kimi was again the only model to expose reviewable prose, but its three attempts reveal why three runs were worth doing: 92, 0, and 99.5.

The 92-point response correctly reconciled the eight-week duration, calculated the compliant cost and benefit, cited the packet, and weighed the confounders. It reached the token ceiling after exposing 7,947 characters, exceeded the word range, and ended abruptly in a malformed citation. The second attempt spent 7,995 of 7,998 completion tokens on reasoning and exposed nothing. The third stopped normally and scored 99 and 100 from the two blind reviewers, producing the final 99.5. It corrected the arithmetic, separated modeled loss from observed outcomes, considered the outage and design weaknesses, and closed with concrete go/no-go gates.

Qwen and DeepSeek produced no visible memo in any of their six attempts. This leaves us unable to compare their potential prose quality under a more generous answer budget. What we can compare is successful handoff: Kimi delivered two useful memos; the others delivered none.

Task 5: editorial writing

The last prompt asked for a 900–1,100-word Kingy.ai analysis of a fictional AI debugging tool. It required the model to calculate true monthly entry prices, audit a vendor’s “2x faster” claim, disclose benchmark exclusions, distinguish Apache-2.0 clients from a proprietary hosted engine, explain data handling, and give reader-specific advice without pretending to have tested the product.

Kimi returned three visible articles and scored 92, 98, and 90. Its best run met the requested form and was judged publication-ready apart from one minor unsupported superlative. The 92-point run was strong on facts and interpretation but exceeded the word limit by more than 10%. The remaining output also missed headline and body-length constraints and used two phrasings that went beyond the packet.

That 90-point article was the only adjudicated response in the benchmark. The two initial AI reviewers scored it 90 and 87, but one raised the possible-invention flag. A third blinded AI reviewer examined the same candidate and set the final score at 90, specifically noting that “vendor-selected” and a claimed field confirmation were not established by the evidence.

Qwen and DeepSeek again reached the limit with reasoning but no article text. It would be wrong to conclude that they cannot write. Under this one-shot 8K protocol, they did not finish the job.

The operating result matters as much as the scores

Across the whole run, only 23 of 45 responses contained visible answer text. There were 22 normal stop finishes and 23 length finishes. Those counts differ because one length-limited Kimi memo still contained a substantial visible answer.

Qwen’s nine blanks map exactly to code repair, synthesis, and editorial writing. Every one reported 8,000 completion tokens, of which 7,998 were reasoning. DeepSeek’s 12 blanks map to every task except constraint reasoning; each reported 7,998 completion tokens, all classified as reasoning. Kimi’s two length finishes were both on evidence synthesis: one visible and useful, one blank.

The practical lesson is simple. A successful HTTP response and a large reasoning trace are not successful task completion. Production monitoring should track visible-answer rate and finish reason alongside latency, token consumption, and cost.

The latency figures are end-to-end wall times. They include OpenRouter, the provider, load or queueing, reasoning, and transfer. With only 15 calls per model, and with some models stopping normally while others exhausted the cap, they are not clean measures of inference speed or tail latency.

Pricing and what the test actually cost

The official-provider list prices differed by more than an order of magnitude:

Sources: QwenCloud pricing, Moonshot Kimi K3 pricing, and DeepSeek pricing. Qwen’s explicit-cache figure is vendor context; this OpenRouter Chat Completions test did not select that cache mode. Fresh stateless requests can still receive automatic prefix-cache hits, so “new conversation” does not mean “uncached.”

The 45 scored calls cost $1.5346001872, reconciling to the dedicated key’s usage delta within about five-billionths of a dollar. Kimi accounted for $0.9046278, or 58.9%; Qwen used $0.599846, or 39.1%; DeepSeek used $0.0301263872, or 2.0%. Repricing the recorded prompt and completion counts entirely at the three vendors’ uncached list rates produces an estimated $1.57699604 for the scored run.

Excluded paid configuration pilots cost another $0.2141522. Two DeepSeek 404 pilots were uncharged. Total API spend for the project was therefore $1.748752382, about $1.75. The $20 account top-up was funding, not usage, and is not counted.

Before execution, a deliberately conservative plan used an 8,000-token input floor or prompt byte length when larger, plus the full 8,000 generated tokens for every cell. It estimated $3.19186464. That was a planning number, not the hard cap. The actual control was a dedicated, non-resetting $10 OpenRouter key limit, exact-route maximum prices, per-cell accounting, and end-of-run reconciliation. The key did not include BYOK spending in its limit, so the run also verified that no BYOK credentials were configured and every response recorded is_byok: false.

DeepSeek’s low spend is real, but price per generated token is not price per useful completion. It spent only two cents on the dollar relative to the total and returned visible answers in three of 15 calls. Kimi was the most expensive route and delivered nearly all the usable work. The right unit depends on the workflow.

A data-policy disclosure

The successful requests used OpenRouter’s provider.data_collection: "allow" setting. The prompts were original synthetic benchmark material and contained no private user data. The account permission for paid endpoints eligible to train on request data was temporarily enabled so the exact pinned DeepSeek route could run, then restored to off and reload-verified immediately afterward.

OpenRouter defines these controls as eligibility: an allowed endpoint may store data non-transiently or train under its own policy. This does not establish that OpenRouter or any provider retained these prompts or trained on them (OpenRouter data-policy routing, provider logging policies).

Audit trail

A redacted companion audit bundle was prepared with this article. It contains 613 allowlisted source artifacts from the completed run: exact prompts and requests, raw responses and retained reasoning fields, endpoint snapshots, returned provider and version metadata, cache and token usage, wall latency, generation cost, deterministic validation reports, blind review packets, reviewer and adjudicator inputs, the post-analysis identity mapping, and the locked aggregation files.

The exporter ran only after all 45 cells and the unblinded analysis were complete. It excludes the API key, raw account and key snapshots, credential fingerprints, the blinding master secret, and local account identifiers. Redactions are listed in PROVENANCE.json, and SHA256SUMS covers every bundled payload plus the generated README and provenance record. The package makes the result inspectable without turning a benchmark download into an account-data leak.

Which model should you use?

Choose Kimi first for mixed one-shot work under this literal high-reasoning, 8K envelope. It was the only model that delivered executable code and reviewable prose, and it was strong when it did. Still, enforce output schemas and inspect length finishes: one JSON fence caused an operational zero, and one of three synthesis runs was blank.

Use Qwen confidently for compact structured tasks, but validate the budget before broad deployment. It was perfect across all six visible constraint and extraction runs. For code or long-form analysis, test a lower or explicit reasoning budget, more answer headroom, or a different supported reasoning level before trusting the route. Qwen’s zeros here reflect failure to surface an answer, not proof that it could not solve the underlying tasks.

Treat DeepSeek as a compelling low-cost specialist, not a drop-in winner under this configuration. It delivered three perfect constraint answers for a fraction of a cent each. On four other task families, native high reasoning plus the 8K shared ceiling yielded 12 blanks. Test lower reasoning effort or a larger completion allowance, and require non-empty content before marking any call successful.

For all three, log the returned model and provider, finish reason, visible content, completion count, reasoning detail, latency, and cost. A retry policy should distinguish invalid formatting from an empty answer, because the corrective prompt and expected cost are different.

Limitations

This is a micro-benchmark: five English-language, text-only, one-shot tasks, one prompt per task family, and three fresh completions per model. It has no confidence intervals and makes no statistical-significance claim. A median based on three values is descriptive, and small score gaps should not be overread.

The 8,000-token total completion envelope was intentionally restrictive and included reasoning. It became the dominant result. Different answer reserves, reasoning controls, direct APIs, or larger limits could change the ranking. Normalized high does not equal matched hidden compute.

We tested hosted OpenRouter endpoints: Alibaba, Moonshot AI MXFP4, and DeepSeek FP8. We did not isolate the local weights. Provider quantization, system prompts, cache behavior, traffic, regional routing, and safety layers may affect the outcome. Cache behavior was recorded but not experimentally isolated.

The suite did not test vision, video, tools, agents, long-context retrieval, multi-turn recovery, self-hosting, security, non-English work, or million-token workloads. Strict contracts also matter: a plausible fenced JSON response can correctly receive zero in a machine interface.

Finally, the two writing tasks were scored by blinded AI reviewers, not humans. Blinding reduced model-identity bias but did not eliminate grader-model preferences or rubric interpretation error. Model availability, route identity, lifecycle labels, prices, licenses, and documentation can change; all should be refreshed immediately before publication.

FAQ

Did Kimi K3 win this comparison?

Kimi won the defined protocol. It returned visible answers in 14 of 15 runs, posted or shared the highest median on all five tasks, and was the sole leader on three. It also handled the shared reasoning-and-answer budget far more reliably than the other two routes. The claim should stop there. Five prompts and three attempts per prompt cannot establish a universal model ranking, and this test did not compare every supported reasoning level, token budget, provider, or direct API.

Why did a blank HTTP 200 response receive zero?

The benchmark scored the deliverable, not the transport. A code repair with no code, a JSON extraction with no JSON, and an article with no article cannot be validated or used, even if the response contains a long reasoning field. The raw response remains in the audit record, and the harness distinguishes a validly routed blank completion from an HTTP or provider failure. That distinction is useful for diagnosis, but both outcomes fail the caller’s task.

Did setting max_tokens to 7,998 break the promised 8,000-token cap?

No. The requirement was a ceiling of 8,000 generated tokens including reasoning. An excluded Qwen pilot requested 8,000 and was reported as 8,002 completion tokens because two other completion tokens appeared beside 8,000 reasoning tokens. The final request used 7,998, then rejected any OpenRouter or provider-native completion count above 8,000. Qwen’s accepted cap hits were reported as exactly 8,000, including 7,998 reasoning tokens.

Why use AI reviewers for the writing tasks?

Code, constraints, and extraction had deterministic answers, so scripts scored them. The two writing tasks required judgments about evidence discipline, arithmetic, decision quality, attribution, structure, and editorial execution. Two separately prompted AI reviewers scored model-blinded packets against frozen rubrics; a third blinded AI adjudicator handled the one triggered case. This improved consistency and reduced model-name bias, but it did not create a human preference study. The article reports that limitation plainly, and the audit package includes the score inputs and outputs.

What should a follow-up evaluation change?

A useful follow-up would separate reasoning and answer headroom instead of letting internal reasoning consume the entire completion allowance. It could compare lower or explicit reasoning budgets, add a larger cap condition, include more prompts per task family, increase repeats, and test direct-provider APIs beside the pinned OpenRouter routes. Those would be new experiments. This result does not predict which adjustment would help each model most.

Did the project stay under budget?

Yes. The 45 scored calls cost $1.5346001872. Paid configuration pilots brought all API usage to $1.748752382, about $1.75. The $20 credit purchase funded the account and is not API consumption. The dedicated key’s non-resetting $10 cap had $8.251247618 remaining after the run.

Bottom line

Kimi K3 won this limited test because it usually left enough room to answer.

Qwen3.8-Max showed a narrower but impressive result: whenever it surfaced output, it was perfect. DeepSeek-V4-Flash-0731 showed that an exceptionally cheap model can still be expensive operationally when a shared reasoning budget repeatedly ends before the handoff.

The most transferable finding is the gap between a valid API response and a usable answer. In reasoning-model deployments, budget management is part of reliability. Measure it that way.