Kimi K3.1 Pro in Our Five-Model Test: Useful Patches, Incomplete Handoffs

Tested October 23, 2026. Kingy ran 40 coding attempts, 40 original research attempts and a separately registered 20-attempt research recovery round. Five API smoke checks were additional. Results below describe our selected tasks, provider routes and agent settings.

Kimi K3.1 Pro produced four saved patches that passed every required test in eight coding attempts. All four runs exhausted their allowance before the agent formally finished the task.

That distinction is the main finding. A model can leave useful code behind and still fail to complete the workflow you asked it to handle. GPT-6.1 Sol left five passing patches, Claude Sonnet 5.5 four, and Kimi K3 and DeepSeek V4.1 Flash 1017 none. Across the group, only two of the 13 passing patches also came from normally completed runs. Both were Claude’s.

K3.1 Pro earned further testing on supervised repository repairs: it passed both attempts on two of our four issues. The research comparison was less conclusive. Our first configuration delivered no answers, and a larger recovery round still exposed output limits that interrupted drafts. We cannot turn these results into an overall model ranking.

For published specifications and wider benchmark coverage, see our Kimi K3.1 Pro guide. This article reports Kingy’s own experiment. A public audit download contains the settings, case identifiers, prompts for the research tasks, result tables, evaluation summaries and scoring disclosures.

The coding test: four issues, two attempts each

We selected historical issues from SWE-bench Verified: unused annotations in Django count queries, chained-exception serialization in pytest, DataFrame dtype preservation in scikit-learn, and class-property documentation in Sphinx.

Each model received the issue, a clean checkout and a bash tool. It could inspect files, edit code and run tests. Reference fixes and evaluator tests were kept outside its environment. Both attempts count; we did not select the better of two.

Before scored calls, we confirmed that each original checkout failed all designated new regression tests and passed its preservation tests. The reference fixes passed all 351 required test outcomes across the four environments. We then evaluated every saved patch in a fresh environment. All 40 evaluations completed without an environment error.

Scroll the table sideways on a small screen.

“Also finished normally” requires both a passing patch and a formal task submission. Costs include all eight coding attempts, including failures, but exclude research and smoke checks. These are selected-case results, not an official SWE-bench score. Four Kimi K3 attempts were interrupted by our accounting check, as explained below.

K3.1 Pro passed pytest and scikit-learn twice each. It left no patch in either Django or Sphinx attempt before the allowance ended. GPT passed pytest and scikit-learn twice, plus one Sphinx attempt. Claude passed scikit-learn and Sphinx twice. Nobody passed Django.

There are four distinct issues here. Repeating them does not create eight independent coding problems, and the one-attempt gap between GPT and K3.1 Pro cannot establish a stable advantage across development work.

Two failures worth examining

Claude’s pytest patches show why the requested fix needs preservation tests alongside it. Both passed the two new chained-exception tests. Both broke an existing check: an unknown serialized entry type should raise an error, but no longer did. Each patch passed 122 of 123 preservation tests and therefore failed the complete-repair criterion. K3.1 Pro and GPT passed all 125 required tests on that issue in both attempts.

Kimi K3 exposed a different problem. In four coding runs, a call reported 4,099 completion tokens against a requested maximum of 4,096. Our strict accounting check stopped the run, although the charge remained inside its money reservation. That is an accounting compatibility failure in this experiment. Its 0/8 includes those interrupted runs and is especially unsuitable as a general capability claim. We retained the failures and did not replace them with coding reruns.

Eleven of the 13 passing patches came from runs stopped by a limit. A test-passing artifact is useful evidence, but those eleven runs did not demonstrate a completed unattended handoff.

Research first tested the limits of our setup

The four research tasks asked for a Kimi launch-claims audit, a five-model API purchasing calculation, NVIDIA financial reconciliation, and a version-pinned Next.js migration plan. Each model received identical frozen local sources for a given task. Across the packets there were 27 source entries from 24 unique primary-source URLs, plus a fictional application fixture. Models could search files and calculate locally; they could not browse for more sources.

The original allowance was six responses, 50,000 cumulative input tokens and 6,000 cumulative output tokens per attempt. None of its 40 attempts saved an answer. Twenty-nine stopped at our conservative input-reservation check, ten reached the response limit, and one encountered a provider error. Missing answers provide no basis for claiming zero factual accuracy.

Our preflight had validated tools, source packets, reference coding fixes and basic model tool calling. It had not demonstrated delivery of a complete research memo from every model within that allowance. A broader comparison needs that calibration on separate tasks before its evaluation settings are frozen.

We registered a recovery round after seeing the original failures: all four research tasks, all five models, one fresh attempt each. It allowed 12 responses, 200,000 cumulative input tokens, 12,000 cumulative output tokens and 16 minutes. We also corrected the output-accounting assumption by requesting at most 4,080 tokens per response and reserving a 16-token margin. Every reported token still counted against the output and spending caps.

Because several conditions changed together, this adaptive round cannot isolate what caused an improvement. It is reported separately and does not replace the original failures.

Scroll the table sideways on a small screen.

The preset threshold was at least nine of ten units in each of correctness, citation support and scope, with no specified critical error. Ten memos were saved; six met it. Four qualifying memos also came from normally submitted runs. Meeting the threshold can still leave a memo needing editing.

These are AI-assisted judgments by Codex, an OpenAI assistant, with no independent human or independent-model review. Review cards removed model names, but the operator also saw run progress, so the assessment was not fully blinded. One contender was an OpenAI model.

A Claude financial memo also sits on a scoring boundary. It gave calculated growth as 105.8%, just outside the frozen 0.05-percentage-point tolerance around 105.8511%, alongside the correct company-rounded 106%. Accepting that companion integer as sufficient would move the memo over the threshold and Claude to two qualifying memos. The audit records both the strict score and this alternative. The counts are too sensitive to delivery conditions and grading decisions to support a research ranking.

What the saved research answers revealed

K3.1 Pro’s Kimi audit satisfied all ten requirements, including access restrictions, unchanged cache-read pricing and attribution of vendor speed claims. Its financial memo also met the threshold.

Its migration plan retained Edge middleware correctly, but prescribed the nonexistent exact npm release typescript@5.1.0. It also selected webpack for production without preserving the custom loader in development. Our registry check returned 404 for that package version; TypeScript 5.1.6 existed. A documented minimum version is not necessarily an installable release pin.

GPT’s migration plan selected TypeScript 5.1.6, used webpack in both development and production, and identified missing fixture components as blockers to executable validation. These were planning exercises. No application was upgraded or deployed.

Delivery limits still interrupted potentially useful work. In three recovery attempts, long bash arguments containing drafts ended as unterminated JSON at the response-token boundary. Those commands could not execute and save the answer. The raw partial drafts remain in the private evidence. The larger cumulative allowance did not remove the roughly 4,096-token per-response accounting envelope.

What the experiment cost

The follow-up consumed US$10.65 in metered API credit, including smoke checks, research recovery and failed requests with reported costs. We matched 757 available generation receipts to recorded charges. One failed request had no retrievable generation receipt, so we kept its US$0.0473825 reservation.

Adding that unresolved amount and a 100% allowance for attributable fees, taxes and uncertainty gives a conservative US$21.40 budget figure. It is not an invoice for US$21.40 of charges. We used existing prepaid credit and local containers, with no credit top-up or rented cloud machine. OpenRouter charges its fee when credits are purchased, so it should not be counted again as a new fee on each request.

Costs here describe our configuration. The common adapter sent no explicit cache-control markers. Several providers recorded automatic prompt-cache hits; Claude recorded none in the original run. We did not reset caches between repeats. These figures cannot establish each provider’s best price, an uncached price comparison, or the cost of an equivalent successful production job. Our Kimi API cost explainer covers those billing distinctions separately.

Settings and evidence that should travel with the results

All five endpoints used OpenRouter with provider fallback disabled and reasoning.effort=high. Routes were Moonshot AI moonshotai/mxfp4 for both Kimi models, OpenAI openai, Anthropic anthropic, and Fireworks fireworks. The exact DeepSeek model ID ended in deepseek-v4.1-flash-1017. A common effort label does not promise equal internal reasoning across providers.

Coding attempts had 12 model responses, 110,000 cumulative input tokens, 12,000 cumulative output tokens including reasoning, and 20 minutes. Cumulative input includes repeated conversation context on each request. A conservative pre-request reservation could stop a run before the nominal token allowance was consumed. The runner used mini-SWE-agent 2.4.6 and fresh Linux containers on a local Apple Silicon machine, with one CPU, 2 GiB RAM and no network per task.

The four coding issues are old and public, so training exposure is possible. We did not add a new private boundary-test suite or test production traffic, security, a working JavaScript upgrade, or reliable use of a million-token context. Timings include local tools, emulation and concurrent work and do not support a controlled speed ranking.

The audit download supplies the frozen protocol hash, model routes, research prompts, coding case IDs and commits, evaluation summaries, scoring tables and cost reconciliation. It is a summary audit package; account snapshots, third-party source copies, raw provider traces and unfinished drafts remain private. The core protocol SHA-256 is 2bdea400f80fa52e04bd06aa6427f2dca495a6d67af3e7fe765a3a178b7b4f28.

For the next coding study, we would use more distinct, recent issues and calibrate full task delivery before freezing the test settings. K3.1 Pro’s pytest and scikit-learn patches give us specific work to examine and extend. Every candidate still needs both code verification and a check that the agent completed the requested handoff.