Editor’s note (September 13, 2026): This article was substantially corrected after an evidence review. We withdrew the unsupported numbered ranking, corrected model identities and benchmark results, and removed case-study claims that lacked traceable evidence. The original publication date remains unchanged.
The best coding model for your team is the one that produces useful, reviewable changes in your codebase at a cost you can sustain. A leaderboard can help identify candidates. It cannot tell you how much debugging or review a particular model will leave for your developers.
This historical comparison revisits five model families covered in the original May 2025 article. It uses specific releases and published evidence available for those early-2025 candidates, with later documentation consulted to verify their identities. It is not a complete survey of every model available then or a current buying guide.
The previous numbered ranking has been withdrawn. The underlying results did not establish a comparable test across the five candidates, and the article did not provide evidence for its claimed business outcomes. This correction uses attributed developer results and editorial evaluation advice; no independent Kingy coding benchmark was run for it.
The five candidates, identified properly
| Candidate | Release or family covered | Scope of this comparison |
|---|---|---|
| Google Gemini | Gemini 2.5 Pro Experimental, March 2025 announcement | That release’s capabilities and reported result; later Gemini updates are excluded |
| OpenAI GPT-4.5 | GPT-4.5 Preview, February 2025 release | GPT-4.5 only; the combined “GPT-4.5/5” label has been removed |
| Anthropic Claude | Claude 3.7 Sonnet, February 2025 release | Its documented model modes and evaluation conditions |
| Meta Code Llama | Code Llama family documented in the 2023 paper and January 2024 revision | A family requiring a size and variant choice, rather than one scored model |
| DeepSeek | Original January 2025 DeepSeek R1 | Full R1; smaller distills and later revisions must be evaluated separately |
Release references: Google, OpenAI, Anthropic, Code Llama paper, DeepSeek model card.
Read the benchmark conditions before comparing scores
SWE-bench evaluates patches for issues in software repositories. Verified is a particular dataset within that project. Results depend on the system around the model, including which files it can inspect, whether it can execute tests and how many attempts it gets. SWE-bench documentation.
The following table records developer disclosures. It is not a leaderboard: the runs were not standardized across vendors, and one uses a smaller task set.
| Model | Published SWE-bench result | Condition that must stay attached |
|---|---|---|
| Gemini 2.5 Pro Experimental | 63.8% on Verified | Google’s custom agent setup, reported in March 2025 |
| GPT-4.5 | 38.0% on Verified | OpenAI labels the coding figures as its best internal performance |
| Claude 3.7 Sonnet | 63.7%; 70.3% with additional computation | Both figures concern the same 489-task subset; the higher result uses parallel attempts, filtering and selection |
| Original DeepSeek R1 | 49.2% resolved on SWE Verified | DeepSeek’s own reported evaluation; not a matched run against these other systems |
| Code Llama | No comparable result retained here | The reviewed paper’s HumanEval/MBPP results cannot fill this column |
Sources: Google’s coding disclosure, OpenAI’s evaluation appendix, Anthropic’s SWE-bench appendix, DeepSeek’s evaluation table, Code Llama paper.
Anthropic’s subset matters: its appendix explains that 11 of the 500 Verified tasks did not work on its infrastructure. The 70.3% figure should not appear as an unconditional score for a standard single attempt. Small numerical differences between rows also do not establish a reliable ordering.
What each candidate brought to an evaluation
Gemini 2.5 Pro Experimental
Google’s March release documented a one-million-token context window and multimodal inputs. Those features make it a reasonable candidate to assess on work involving substantial code context or visual specifications. The inference is about what to test; it does not establish better patch quality on your repository. Google’s release announcement.
The original article’s near-perfect coding claims have been removed. For a repository task, measure whether supplying more context improves the result and whether the additional processing time is worthwhile.
GPT-4.5 Preview
GPT-4.5’s documented context window is 128,000 tokens. The combined GPT-4.5/5 entry and million-token claim were incorrect. The documentation now marks GPT-4.5 Preview as deprecated, so this section should be read as historical coverage. GPT-4.5 specifications.
OpenAI introduced it as a research preview. Its launch evaluation supports the result in the table above; it does not substantiate the original article’s 85% claim. GPT-4.5 launch.
A model name also does not establish which model an editor extension uses. Check the selected model and the integration’s documented capabilities before describing a workflow as tested or supported.
Claude 3.7 Sonnet
Anthropic introduced standard and extended-thinking modes alongside a limited research preview of Claude Code. These are relevant distinctions for testing interactive assistance versus an agent that can inspect and modify a repository. Claude 3.7 Sonnet launch.
Record the mode, permitted tools and thinking budget when evaluating it. The earlier description of Claude as simply the budget option was not supported by a consistent cost comparison. A useful measure is the total cost of reaching an acceptable patch, including failed attempts and review time.
Code Llama
Code Llama is a family with base, Python and instruction-following variants. The paper describes training on 16K-token sequences and evaluations extending to 100K inputs; it does not support assigning every variant a universal 128K limit. Code Llama paper.
Select an actual size and variant before testing. The model’s downloadable weights also come with license terms; the relevant release uses a Llama community license. Code Llama checkpoint license.
This can be an evaluation path when control of the deployment is a requirement. Hardware, maintenance and reviewer time still belong in the budget. No evidence retained here establishes Code Llama as the leading free coding model of 2025.
DeepSeek R1
Original R1 is a full reasoning model, with smaller Qwen- and Llama-based distills listed separately. A local distill’s performance should be reported under its own name. DeepSeek’s model and distill documentation.
For debugging, test whether the proposed diagnosis leads to a correct patch. A convincing explanation can still miss an edge case or recommend an incompatible library. Record the time required to review and correct the output rather than assuming that a longer explanation saves work.
A useful comparison starts with your own tasks
Use a small set of representative tasks to screen candidates. The examples below are a proposed method, not case studies or measured outcomes.
| Task | What to inspect |
|---|---|
| Fix a reproduced bug | The regression test passes and existing behavior remains intact |
| Add a feature across several files | The change follows repository conventions and avoids unrelated edits |
| Update a dependency | The generated code uses the target version’s real interfaces |
| Turn a UI specification into code | The interface matches the specification and works at the required sizes |
| Explain unfamiliar code | The explanation matches the implementation and identifies uncertainty |
Run candidates against the same starting repository and acceptance criteria. Record the exact model, application or agent, settings, tools and attempt limits. If one configuration gets extra computation or retries, include that difference in the results.
Track three outcomes: whether the change was accepted, how much human review and repair it required, and its total operating cost. Keep failed attempts in the record. Use the initial trial to narrow the field, then validate the preferred configuration on additional tasks before making a broader commitment.
The original anonymous fintech and startup stories have been removed because their claimed uptime and development-cycle improvements lacked traceable evidence. A useful case study would identify the workload, baseline, observation period and measurement method. Until those records exist, use explicitly hypothetical examples for planning and reserve performance claims for results you can substantiate.
The Kingy Brief
Get the next Kingy Brief.
Source-checked AI changes, original tests and one practical thing to try.
Free · Choose your subjects · Double opt-in · Unsubscribe anytime