DeepSeek V4.1 Flash vs GPT-6, Claude and Kimi K3.1 Pro: 32 Trials + Cost

Your decision

The model routes and tasks you want to compare.

What to compare

Compare outcomes, failures and costs from 24 trials.

Evidence

Tested September 10, 2026. The small sample and shared runs are explained in the original methods.

All spending includes unsuccessful attempts. Cost per passing trial divides the total model bill by its number of passing trials. The median time sums HTTP response waiting time within each task, including network and provider processing; it excludes local grading and billing lookups. It is not a throughput or time-to-first-token benchmark.

All four routes passed both coding tasks on both attempts. Kimi also passed both chart and ledger trials. The earlier differences came from the small integration tasks: DeepSeek missed one ledger run, and Claude missed the strict formatting requirement on both chart runs. Eight attempts per model are too few to turn those counts into a general reliability ranking.

DeepSeek’s second ledger trial returned {"cursor":"null"}, supplying a string where the first-page request needed JSON null. It repeated that cursor after an error and finally tried an empty string. It never read a page or produced a final answer within four responses. Its first attempt completed the same workflow successfully.

That trace supports a narrow finding about the tested gateway route. We did not isolate every model and adapter component or test whether more turns would recover the job. The task stopped at the limit fixed before the run.

Claude returned the correct chart values on both attempts. Each answer also included explanatory prose before its JSON object, despite the prompt asking for only JSON. We counted both attempts as failures under the predetermined output contract. The first response was:

Eligible: Cobalt (7 leads, $84 → $12/lead), Amber (10 leads, $90 → $9/lead), Jade (8 leads, $64 → $8/lead).

{"eligible_names": ["Amber", "Cobalt", "Jade"], "winner": "Jade", "total_spend_usd": 238, "total_qualified_leads": 25, "blended_cost_per_lead_usd": 9.52}

That distinction matters. A human could read the answer and understand it. Our application contract required json.loads(answer) to work on the entire response. A prefatory sentence breaks that contract even when the following object is correct.

We tested instruction-following on ordinary chat completions. We did not enable provider-enforced JSON schemas or add a repair layer. In an application, those are sensible options to investigate where supported. This result is not evidence that Claude cannot interpret the chart or that its structured-output feature failed.

The first coding task repaired interval merging: preserve inputs, combine overlapping or touching windows, keep nested ranges intact, discard empty intervals and reject invalid ones. Nine checks assessed each answer.

The second repaired proportional discount allocation. It required exact integer arithmetic, deterministic tie-breaking and correct behavior above JavaScript’s familiar 2⁵³ precision boundary. Thirteen checks assessed each answer. Passing these functions is useful; it is a much smaller demand than resolving an unfamiliar repository issue.

The tool task required recovering from a retryable mock error, reading two ledger pages, deduplicating an invoice and excluding void or refunded records. Every operation stayed inside a synthetic fixture. It tested a short recovery sequence, not a long autonomous session.

The official V4.1 table compares with GPT-5.6 Sol and Claude Opus 5.0, among other models. Those are different opponents from the GPT-6 Astra and Claude Fable 5.1 endpoints in our September 10 test. Do not silently substitute the newer names into the launch chart.

Source: DeepSeek’s launch model card at revision df42c109. These are vendor-reported scores, not Kingy reproductions. The HLE row uses the main score, not DeepSeek’s separately reported text-only subset. DeepSeek used maximum reasoning effort for its instruct evaluations; its agentic results also depend on the stated harness.

The table supports a specific claim: V4.1 is competitive on some agentic evaluations at its advertised token prices. It also shows substantial deficits on other evaluations. A small numerical lead on one benchmark does not establish a statistically meaningful general advantage.

Different benchmark versions answer different questions. Leading Terminal-Bench 2.1 while trailing Terminal-Bench 4.0 is a reason to inspect task difficulty and setup. Averaging those percentages with a knowledge exam would produce a convenient number without a clear meaning.

The original price rows below describe the September comparison. Kimi’s added row was checked October 23. These list prices are separate from the measured trial bills.

Sources: DeepSeek pricing, OpenAI’s GPT-6 Astra documentation, Anthropic’s model overview, the gateway catalog and Vercel’s Kimi K3.1 Pro listing. Cached input, regional routing, long context and service tiers can change the bill. Our requests used short contexts and pinned standard provider routes.

In our sample, GPT’s cost per passing trial was approximately 7.7 times DeepSeek’s, and Claude’s was 20.0 times DeepSeek’s. Those ratios include each model’s failures and its actual token usage. They apply to these short tasks and settings; they are not savings forecasts for a production agent.

Kimi’s October total was $0.0331362 across 14 requests, or $0.004142025 per passing trial. All generation receipts identify Moonshot AI and reconcile to the dedicated key’s spend. Existing Gateway credit covered the run; no new cash charge or top-up occurred. The test key was revoked after reconciliation.

Reasoning effort deserves particular care. All models received the gateway’s low effort setting and a 3,072-token output cap. That does not provide equal internal compute. Reported reasoning-token fields also differ by provider; a zero in the returned usage field does not prove that a model performed no internal reasoning. The billing records are the stronger evidence for this cost comparison.

DeepSeek publishes downloadable V4.1 Flash weights under MIT. GPT-6 Astra and Claude Fable 5.1 are hosted proprietary offerings in this comparison; their model weights are not supplied for a comparable self-hosted installation. Open weights, an API and a fully reproducible training process are different things. The presence of a weight download alone does not establish complete training reproducibility.

For Kimi K3.1 Pro, Vercel’s October 23 announcement says Moonshot plans to release weights by November 4. This extension tested only the hosted API; it provides no local-hardware or self-hosting evidence. Vercel launch announcement.

V4.1’s own checkpoint is large: our audit found about 510.3GB across 48 official weight files. Its attractive API price does not imply that you can run the official release on a laptop. See the V4.1 local hardware guide for the RAM, VRAM and runtime evidence.

For a smaller model you can evaluate for desktop deployment, our DeepSeek versus Qwen3.8-27B comparison uses the same task campaign and explains the difference between hosted results and local hardware claims.

For these short tasks, DeepSeek had the lowest recorded bill and one ledger failure. Kimi completed all eight trials for $0.03314; GPT-6 Astra also completed eight and had a shorter recorded median wait. Kimi is a useful candidate when strict JSON and a short tool workflow matter, but eight trials across two test dates cannot predict production reliability or speed. Claude’s two format misses remain a reason to validate the output contract for this workflow.

For difficult repository work, long research sessions or ambiguous decisions, these four fixtures are insufficient. Use the published benchmarks to choose candidates, then evaluate a few representative tasks with an acceptance test. Include retries, tool rounds and human review time in the economics.

A cheap model that fails your task can be expensive to operate. A premium model that provides no additional value on a simple extraction can also be expensive. The useful unit is a correctly completed job under your application’s rules.

We fixed the four prompts, fixture image and checks before the September run. Each model received two fresh attempts per task. Provider order was reversed for repeat two in that campaign. Sampling stayed at provider defaults. The October Kimi extension used the identical system message, prompts, chart bytes and grader, with the same low reasoning setting, 3,072-token output cap and four-response tool limit. The only tested model in the extension was moonshotai/kimi-k3.1-pro, pinned to moonshotai. No provider or model fallback was configured. Equal effort labels do not establish equal internal compute.

Each tool trial allowed at most four model responses, including the final answer. There were no paid retries to improve a failed model answer, no answer repair and no LLM judge. All response statuses, finish reasons, charges and outputs are retained.

The downloadable evidence bundle contains the complete 32-trial campaign, including Qwen, with exact requests, responses, grading and billing records. The original four-model September campaign used 56 model calls and cost $0.313674 in confirmed gateway charges, well below our $10 limit. No GPU rental or paid judge was required. Inspecting the files is free; rerunning the API script incurs usage charges.

The October 23 Kimi extension adds eight trials and 14 calls for $0.0331362, within its $2 existing-credit reservation. Both ledger trials used all four permitted responses: retryable mock error, first page, second page, final JSON. All calls returned HTTP 200, with no truncation or paid rerun. The run and deterministic grading were performed by Codex for Kingy. The retained preflight records matching prompt, grader and image hashes.

GPT-6 Astra passed all eight trials and had the shortest median API waiting time. DeepSeek V4.1 Flash passed seven at the lowest cost. Claude Fable 5.1 passed six: it read our chart correctly twice, but added prose that broke the requested strict JSON format.

We tested DeepSeek V4.1 Flash, GPT-6 Astra and Claude Fable 5.1 on four small tasks, with two runs of each task per model. The results show useful cost and integration differences. They do not reproduce DeepSeek’s launch benchmarks or settle which frontier model is best.

Tested September 10, 2026. These 24 trials are a subset of one 32-trial campaign that also tested Qwen3.8-27B. The DeepSeek runs are shared with our Qwen comparison.

Browse Kingy Tested articles to compare the retained evidence by model, vendor and year. The DeepSeek comparisons share the September 10 campaign; this article also includes a separately dated Kimi extension. The shared DeepSeek trials are not independent replications. Opening the saved bundle makes no model calls.