If you want the short answer, DeepSeek V4 Flash is the only category winner on this page that Kingy can currently call tested. In a predeclared 20-task repository trial, it produced 19 accepted fixes, compared with 17 for the normal Codex model, while taking almost twice as long.
The other categories are deliberately marked candidate, not winner. Qwen3.8-27B, GLM-5.3-Flash, Mistral Small 4, Devstral Small 2, and other recent releases have strong official specifications, but a vendor benchmark is not a Kingy test. This leaderboard changes when the evidence changes.
Searchers often call all downloadable models “open-source LLMs.” That is too loose. Most models below are more accurately described as open weight: you can download their trained weights under a specific licence, but the complete training recipe and data may not satisfy the Open Source AI Definition.
Evidence coverage at a glance
| Evidence state | Coverage now | Meaning |
|---|---|---|
| TESTED | 1 controlled category winner | Coding agent: DeepSeek V4 Flash 0731, for the published 20-task cohort only |
| TESTED | 1 measured workload | Cheap hosted API: the same DeepSeek run establishes measured cost, not an exhaustive API winner |
| CANDIDATE | 2 candidate-only categories | Local and multimodal candidates are sourced but have not completed Kingy’s controlled category tests |
| NO WINNER | 1 unresolved category | Long-context claims remain advertised limits rather than a controlled quality result |
| TIE | 1 licence-only conclusion | MIT and Apache-2.0 checkpoints tie at the permissive-licence layer; capability is not implied |
Next planned test: a matched cheap-API comparison using deterministic structured extraction, grounded summarization, and tool-use tasks. No run date has been announced.
Kingy-tested results
| What you need | Current answer | Status | Why | Important limit |
|---|---|---|---|---|
| Coding agent | DeepSeek V4 Flash 0731 | TESTED | 19/20 accepted fixes versus 17/20 for the comparison model in Kingy’s controlled paired cohort; MIT weights; very low metered cost | One Python-heavy 20-task sample; slower than the comparison model; newer GLM and Qwen releases remain untested |
| Cheap hosted API | DeepSeek V4 Flash, among Kingy-tested models | TESTED | Historical run cost $0.3433; same ledger is about $0.727 off-peak at current rates | Tested workload only—not an exhaustive provider comparison; privacy, latency, retries, and peak pricing matter |
Tested evidence lineage
| Model and result | Exact version | Test date and denominator | Evaluation record | Raw evidence | Superseded? |
|---|---|---|---|---|---|
| DeepSeek V4 Flash: 19/20 accepted fixes; comparison model: 17/20 | DeepSeek-V4-Flash-0731 | August 2, 2026 · 20 counted tasks per model | Controlled Kingy coding harness; no standalone evaluator version is recorded in this public package | Complete test · Leaderboard results CSV | No—current for this cohort |
Candidate watchlist and unresolved categories
| What you need | Current answer | Status | Evidence available | Important limit |
|---|---|---|---|---|
| Local model | Qwen3.8-27B is the first candidate to test | CANDIDATE | Official sources: 27B dense, native multimodal, Apache 2.0, 262K native context | Kingy has not completed the controlled local head-to-head; 27B still requires substantial memory |
| Multimodal model | No winner yet; test Qwen3.8-27B, GLM-5.3-Flash, Mistral Small 4 | CANDIDATE | Official sources publish multimodal support and downloadable weights | No common Kingy image/document test yet; deployment sizes differ dramatically |
| Long-context model | No winner yet | NO WINNER | Advertised limits only: DeepSeek, GLM, and Qwen publish million-token-class options | Advertised capacity is not retrieval quality; Kingy has not run the required depth/degradation test |
| Permissive licence | Tie at the licence layer | TIE | Primary licence documents for the named checkpoints | A permissive weight licence does not make the full AI system OSI-open or prove model quality |
On this page
- Evidence coverage
- Kingy-tested results
- Candidate watchlist
- The tested coding winner
- Cheapest measured API workload
- Local-model candidate
- Multimodal candidates
- Long-context status
- Permissive-licence tie
- Full methodology
- Monthly changelog
- Evidence downloads
Best open-weight coding model: DeepSeek V4 Flash 0731
DeepSeek V4 Flash is Kingy’s current coding-agent pick because it is the only leading open-weight model in this group that Kingy has measured in a controlled repository test.
The test used 20 preselected SWE-bench Verified issues, clean pre-fix repository snapshots, hidden defect and regression tests, identical task instructions, disabled network access, and a fixed time limit. DeepSeek accepted 19 of 20 tasks. The normal Codex model accepted 17. DeepSeek made no human-assisted code corrections and introduced no regression in an accepted patch.
It was not faster. Median wall time was 4 minutes 46 seconds versus 2 minutes 24 seconds, and aggregate agent time was 1.93 times higher. DeepSeek also used roughly four times as many input tokens. The result is therefore “more accepted fixes in this sample, for very little money, but substantially slower”—not “best coding model everywhere.”
Read the complete 20-task test or inspect the coding index and public CSV.
What could change this answer
GLM-5.3-Flash arrived on August 26 with MIT weights and new vendor-reported coding results. Qwen3.8-27B is a newer Apache-2.0 candidate designed for coding and long-horizon agent work. Devstral Small 2 remains a practical 24B coding model that Mistral says can run on one RTX 4090 or a 32GB Mac. None has completed the same Kingy test, so none is ranked ahead of or behind DeepSeek yet.
Cheapest open-weight API: DeepSeek wins the measured workload, not the universe
The original August 2 DeepSeek run used 869,195 uncached input tokens, 33,110,912 cached input tokens, and 460,485 output tokens. The actual bill for the 20 counted tasks was $0.3433, or $0.0181 per accepted fix.
DeepSeek changed its prices on August 16. At the current published schedule, the same token ledger would cost approximately:
| Price period | Re-estimated total | Re-estimated cost per accepted task |
|---|---|---|
| Off-peak | $0.727 | $0.038 |
| Peak | $1.454 | $0.077 |
Those are arithmetic re-estimates, not new charges. They preserve the historical experiment while giving a current planning number.
Token price alone is not the decision unit. Measure cost per accepted task after retries, latency, tool calls, human review, and failures. DeepSeek’s test result was cheap partly because its automatic prefix cache absorbed 97.44% of input tokens. Another workload with less reusable history may price very differently.
For comparison, Mistral currently lists Mistral Small 4 at $0.15 per million input tokens and $0.60 per million output tokens. That does not establish equivalent capability, cache behavior, or completed-task cost.
Best local model: Qwen3.8-27B is the next candidate, not a crowned winner
Qwen3.8-27B is the most interesting current general-purpose local candidate in this update. Qwen publishes it under Apache 2.0 as a 27B dense vision-language model with 262,144 tokens of native context and an extension path to one million tokens.
That combination makes it relevant to local chat, coding, document work, and image understanding on a serious workstation. It does not make it the winner before testing. A 27B dense model still has to move all 27 billion parameters; active-parameter claims from sparse models and quantized file sizes are not interchangeable with real generation speed.
For local coding specifically, Devstral Small 2 is the more targeted candidate. Mistral publishes a 24B Apache-2.0 checkpoint and says it is designed for tool-driven repository work. The model card says it can run on a single RTX 4090 or a Mac with 32GB of memory. Kingy has not yet measured it against Qwen3.8-27B under the same quantization, runtime, context, and repository tasks.
The local winner test must record model file, quantization, runtime commit, device, memory use, time to first token, generation rate, quality score, and failures. “It loaded in Ollama” is not an evaluation.
For setup instructions rather than rankings, use Kingy’s complete local AI hardware and installation guide.
Best open-weight multimodal model: no honest winner yet
The current test slate is:
- Qwen3.8-27B: Apache 2.0; image and video understanding; 27B dense.
- GLM-5.3-Flash: MIT; native multimodality; 320B total and 18B active parameters.
- Mistral Small 4: Apache 2.0; text and image input; 119B total and 6B active parameters per token; 256K context.
They are not equivalent deployment targets. Qwen is the workstation-scale candidate; GLM and Mistral’s newest sparse models require much more storage and serving infrastructure even when their active parameter counts look modest.
Kingy’s multimodal test should include receipts, dense tables, charts, screenshots, photographed documents, diagrams, adversarial OCR, and images where the correct response is “the evidence is not visible.” Score exact extraction separately from visual reasoning and hallucination resistance.
Until those results exist, any single winner would be a vendor-benchmark collage.
Best long-context open model: advertised capacity is not the test
DeepSeek V4, GLM-5.2/5.3, and Qwen3.8 all publish million-token-class context options. That tells you how much input an implementation may accept. It does not tell you whether the model retrieves the right passage at 800,000 tokens, follows an instruction introduced near the start, resists distractors, or finishes before the latency budget expires.
Kingy’s DeepSeek coding test reached a maximum single request of 125,580 input tokens. Neither backend compacted or overflowed. That establishes useful headroom in one agent workflow, not a one-million-token quality result.
The long-context category will remain unranked until Kingy runs the same retrieval, synthesis, instruction-retention, and distractor tasks at multiple depths, records end-to-end latency and cost, and publishes failures.
Most permissive open-weight model: treat this as a licence filter
At the checkpoint level, several current candidates use standard permissive licences:
| Checkpoint | Published licence | Practical classification |
|---|---|---|
| DeepSeek V4 Flash | MIT | Permissively licensed open weight |
| GLM-5.3-Flash | MIT | Permissively licensed open weight |
| Qwen3.8-27B | Apache 2.0 | Permissively licensed open weight |
| Devstral Small 2 | Apache 2.0 | Permissively licensed open weight |
| Mistral Small 4 | Apache 2.0 | Permissively licensed open weight |
The licence layer can produce a tie. Do not manufacture a decimal “openness score” to break it.
Also avoid the shortcut “Apache or MIT means the model is fully open source.” The Open Source Initiative says an open-source AI system needs the freedoms to use, study, modify, and share, along with the preferred form for modification and information about the data used to derive the parameters. Downloadable weights under a familiar software licence may still be only part of that package.
For clause-level comparisons, use Kingy’s open-weight licence index. It is a reading of published documents, not legal advice.
How Kingy ranks models
Every category has a separate pass condition because a single composite score would hide the decision:
- Local: quality subject to a fixed hardware and memory envelope, then latency and usability.
- Coding: hidden-test acceptance, regressions, correction rate, agent time, tool failures, and cost per accepted task.
- Multimodal: extraction accuracy, visual reasoning, abstention, latency, and cost.
- Long context: retrieval and instruction retention across depth, plus degradation, latency, and cost.
- Licence: exact checkpoint terms, redistribution, commercial use, restrictions, attribution, and output clauses.
- API cost: accepted work at the required quality, not nominal token price.
Vendor results may place a model on the test slate. They never determine a Kingy winner.
FAQ
What is the best open-source LLM in 2026?
There is no defensible universal winner. DeepSeek V4 Flash is Kingy’s current tested coding-agent pick. Local, multimodal, and long-context categories remain open until the leading candidates complete the same controlled tests.
What is the best open-source coding LLM?
DeepSeek V4 Flash produced 19 accepted fixes in Kingy’s 20-task repository cohort, versus 17 for the normal Codex model. Treat that as the result for one predeclared sample, not a claim about every language, repository, or agent harness.
What is the difference between open source and open weight?
Open weight means trained parameters are downloadable under some licence. Open-source AI is a stronger claim involving the practical freedom and materials needed to use, study, modify, and share the system. Kingy uses the exact checkpoint licence and avoids calling every downloadable model fully open source.
Which open model should I run locally?
Qwen3.8-27B is the next general-purpose candidate Kingy should test. Devstral Small 2 is the targeted local coding candidate. Choose only after matching the quantized model and context size to your real available memory.
Does a one-million-token context window mean the model is best at long documents?
No. It establishes an advertised capacity, not reliable retrieval, reasoning, latency, or cost at that depth. Kingy has not yet named a long-context winner.
Can I use these models commercially?
MIT and Apache 2.0 checkpoints generally allow commercial use subject to their terms, notices, and any third-party rights. Bespoke model licences can add revenue, territory, attribution, use, or output restrictions. Check the exact files for the exact checkpoint and obtain legal advice for consequential deployments.
Sources and disclosure
Kingy paid for the DeepSeek API credits used in its coding test. No vendor selected the tasks or reviewed the result. Specifications and prices in this page are linked to official model cards, licence files, or provider documentation and were rechecked on August 30, 2026. Vendor benchmarks are treated as vendor claims until Kingy reproduces the relevant workload.
Full leaderboard methodology
The rules below govern every current and future category. They are published in full so a winner cannot be changed by an undocumented editorial preference.
Governing rules
- A model may enter the candidate slate from an official release, official model card, or meaningful independent result.
- A category winner requires a controlled Kingy test in that category. Vendor results never determine the winner.
- Exact model version, weights revision, quantization, provider route, and test date are mandatory.
- Tests are predeclared before counted runs. Pilot runs are discarded symmetrically and documented.
- Failures remain in the denominator. A rerun is allowed only for documented infrastructure failure, not for an unsatisfactory answer.
- Raw task outcomes, token ledgers, timing, cost calculation, prompts, and environment data are published.
- Category scores are not combined into a universal “intelligence” number.
- A new release does not automatically displace a winner; it enters the next test slate.
Evidence labels
kingy_tested: completed a controlled Kingy category test with raw outcomes.official_source_only: specifications, prices, or licence verified from a first-party source; capability not tested by Kingy.third_party_result: independent external test with reproducible detail; not a Kingy run.vendor_claim: benchmark or performance claim published by the model vendor.not_verified: unresolved or insufficient evidence.
The page must display the label in human-readable language. Colour alone may not convey it.
Category protocols
Local model
Test profiles:
- 16GB unified memory or RAM;
- 24GB VRAM;
- 32GB unified memory;
- optional 64GB workstation tier.
Within a tier, use the same runtime build, context target, sampling policy, prompt set, warm-up policy, and power state. Record:
- exact weights and quantization;
- model file size and peak resident memory;
- load time and installation failures;
- time to first token;
- prompt processing rate and output tokens per second;
- quality on extraction, instruction following, reasoning, writing, and one tool task;
- crashes, repetition, premature EOS, context shifts, and malformed tool calls.
Winner rule: highest task acceptance subject to fitting the declared tier without swap-induced failure; break a material tie by median latency, then installation reliability. Do not compare different hardware tiers as one ranking.
Coding agent
Reuse the existing predeclared paired-repository design and public coding index. Expand beyond Python over time.
Record:
- hidden-test acceptance and regression count;
- first-attempt acceptance and correction rescue rate;
- wall time, agent time, requests, tool calls, and patch failures;
- uncached input, cached input, output/reasoning tokens;
- actual cost and cost per accepted task;
- harness, tool permissions, timeout, network policy, and reasoning effort.
Winner rule: highest accepted-task count. If statistically tied, prefer lower regression rate, then lower cost per accepted task within the same latency ceiling. Report small-sample uncertainty rather than decimal theatre.
Multimodal
Use at least five task groups:
- OCR with clean and degraded receipts;
- dense tables and multi-column documents;
- chart extraction plus one derived calculation;
- diagrams and visual-spatial questions;
- unanswerable or occluded images requiring abstention.
Score exact fields separately from rubric-based reasoning. Record image resizing, token conversion, latency, and cost. Any model that fabricates hidden content fails the abstention item.
Winner rule: highest macro-average across task groups, provided abstention accuracy and exact extraction each clear the predeclared floor.
Long context
Test at 32K, 128K, 256K, 512K, and the highest mutually supported tier. Include:
- exact needle retrieval at 10%, 50%, and 90% depth;
- multi-hop synthesis using evidence separated across the context;
- an early instruction that conflicts with a later distractor;
- repeated near-match distractors;
- a long repository or document task with a verifiable result.
Record effective context, truncation, compaction, prefill latency, first-token latency, completion time, cache behaviour, tokens, cost, and output quality.
Winner rule: highest accuracy at the highest tier where all compared models complete, subject to a published latency ceiling. A model does not win by accepting a larger request that it answers incorrectly.
Permissive licence
Read the exact repository licence, model card, notices, and incorporated component licences. Record:
- SPDX or vendor licence name;
- commercial use and field-of-use restrictions;
- redistribution and derivative rights;
- revenue, user-count, territory, naming, and attribution triggers;
- output restrictions and model-training restrictions;
- acceptable-use policy incorporation;
- licence-file commit or retrieval date;
- whether the evidence establishes only an open-weight licence or a complete OSAID package.
Winner rule: this is a filter, not a capability score. Standard MIT/Apache-2.0 checkpoints can tie. State “permissive licence tie” and choose the model through the relevant capability category.
Cheap hosted API
Use at least three repeatable workloads: structured extraction, grounded summarization, and a tool-using task. For coding, reuse the coding-agent protocol.
Record:
- first-party price and timestamp;
- cached/uncached input, output, reasoning tokens, tool charges, and batch/peak rules;
- completion rate, retry count, latency, and human-review time;
- actual bill where available;
- cost per accepted task and cost per 100 accepted tasks.
Winner rule: lowest cost per accepted task among models meeting the same quality and latency floor. Nominal token price cannot win the category.
Statistical and editorial safeguards
- Publish counts and confidence limits for binary outcomes when the cohort is large enough; for small cohorts, foreground the denominator.
- Do not turn a 19/20 sample into “95% accuracy” on the whole benchmark.
- Predeclare tie-breakers and minimum practical differences.
- Retain task-level results when a model version changes; never overwrite history.
- A provider-side model update creates a new version row even if the public alias is unchanged.
- Human qualitative scoring requires blind model labels and at least two raters for publication-quality claims.
Monthly cadence
Week 1: source and licence recheck.
Week 2: run one highest-impact missing category or challenger.
Week 3: reconcile raw data, costs, and failure notes.
Week 4: update the canonical table and changelog; no winner change without completed evidence.
Trigger an out-of-cycle review for a model/version replacement, licence change, price change greater than 20%, removed API, disclosed benchmark error, or material test-harness defect.
Monthly changelog
Every material leaderboard change remains visible. Earlier results are retained rather than overwritten.
| Date | Scope | Change | Evidence | Impact |
|---|---|---|---|---|
| 2026-08-28 | leaderboard | Replaced universal winner framing with tested/candidate/unranked evidence states | Editorial audit | Prevents unsupported tested claims |
| 2026-08-28 | coding | Retained DeepSeek V4 Flash 0731 as tested coding leader for the existing 20-task cohort | Kingy controlled test | Coding category has one evidence-backed winner |
| 2026-08-28 | pricing | Added current-price re-estimate after DeepSeek’s August 16 rate change while preserving the historical bill | Official pricing plus published token ledger | Current off-peak estimate is about $0.727 total; peak about $1.454 |
| 2026-08-28 | candidates | Added GLM-5.3-Flash and Qwen3.8-27B to the evaluation slate | Official model cards | No ranking change until controlled tests |
| 2026-08-28 | content-cluster | Consolidated the broad June 21 open-weight shortlist into this canonical living leaderboard | Content audit, Search Console review, and completed publication | Canonical leaderboard published August 30; any competing-page redirect remains deferred pending post-publication data |
Evidence downloads
The public data is provided as stable, versionless CSV files so citations do not depend on JavaScript:
- Download the complete leaderboard results CSV
- Download the monthly changelog CSV
- Download the official-source register CSV
The results file includes evidence states and limitations. The changelog preserves editorial changes. The source register identifies the primary source used for each time-sensitive claim.
The Kingy Brief
Get the next Kingy Brief.
Source-checked AI changes, original tests and one practical thing to try.
Free · Choose your subjects · Double opt-in · Unsubscribe anytime