DeepInfer Explained: DeepSeek’s Open Inference Engine, Benchmarks, Specs, and Costs

DeepSeek has opened the inference engine behind V4.1-Flash. Its benchmarks show why large-scale serving needs specialized software, while its deployment notes reveal how much infrastructure still sits around the engine.

DeepSeek released DeepInfer on October 20, 2026, giving developers access to the inference framework it says powers its production services. The practical significance is access to the engineering that turns model weights into a service: GPU kernels, expert distribution, speculative decoding, scheduling, and output constraints.

The release deserves attention from AI infrastructure teams. It also needs careful reading. The public package provides offline inference, supports DeepSeek-V4.1-Flash on specified NVIDIA and Huawei accelerators, and leaves several service-level components for operators to build. Downloading it does not reproduce DeepSeek’s hosted platform.

Our assessment is that DeepInfer’s greatest immediate value is the combination of usable engine code and unusually specific deployment guidance. The published performance curves are useful reference points. They are insufficient evidence for a universal speed advantage, a guaranteed hosting price, or a measured reduction in electricity consumption.

Research checked October 20, 2026. This article is based on DeepSeek’s public source code, documentation, charts, model card, and API pricing. Kingy.ai did not run DeepInfer, rent accelerators, or independently reproduce the benchmarks. Vendor measurements, chart estimates, and our illustrative calculations are identified below. Repository analysis is pinned to commit 774ce244b87c; later changes may alter support or results.

DeepInfer specs and release scope

DeepInfer is software for executing a trained model. Model capabilities come from the checkpoint and its training; the engine determines how efficiently that checkpoint runs on a particular system. Keeping those layers separate prevents an accuracy score from being mistaken for an engine speedup.

Sources: the release README, engine license, and checkpoint card. The listed hardware support does not establish compatibility with a Mac, consumer RTX card, H100, H200, or a different Ascend generation.

The underlying V4.1-Flash architecture is a 552-billion-parameter mixture of experts, according to DeepSeek’s model announcement. It uses a causal encoder–decoder design with 8 billion active parameters for input processing and 16 billion for output. Those active counts describe computation per token; they do not reduce the entire checkpoint’s storage or residency requirements to those of an 8B or 16B dense model.

The 1017 configuration specifies 40 text layers, 384 routed experts, six selected experts per token, a 128-token sliding window, and 1,048,576 maximum positions. These help explain why expert placement, cache handling, and long-context execution matter. A model’s maximum context remains separate from the 131,072-token context used in the headline engine charts.

How DeepInfer reduces inference work

Kernels, expert distribution, and scheduling

DeepInfer combines the FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect libraries with custom TileLang kernels. These specialize different parts of attention, matrix operations, communication, and selection. Their integration matters because accelerating arithmetic alone can leave an engine waiting on communication or CPU scheduling.

Mixture-of-experts inference creates a second balancing problem. Tokens select different experts, so some accelerators can receive more work than others. DeepSeek’s deployment Q&A describes collecting recent expert loads, replicating busy experts, and periodically changing placement. A relatively small amount of redundant weight storage can help avoid waiting for the busiest rank.

The engine also overlaps CPU scheduling with accelerator execution. While the device processes one step, the CPU prepares subsequent work. DeepSeek calls this a zero-overhead batch scheduler. Read that as an optimization goal for the scheduling pipeline, rather than a claim that a full inference service has zero overhead.

The same Q&A describes low-precision formats for expert weights and cache data. That makes numerical validation part of performance evaluation: deployment teams need to check answer quality and probability behavior alongside throughput, especially when changing kernels or hardware.

DSpark spends verification work selectively

Speculative decoding generates candidate tokens and asks the main model to verify them. It helps when enough useful tokens survive verification to justify the drafting and checking work. Acceptance rates therefore matter as much as a headline draft length.

DeepSeek’s DSpark implementation adds confidence estimates and variable-length verification. Its production description says a draft model proposes five tokens, then the scheduler selects verification work using expected acceptance and a profiled hardware throughput curve. Low-confidence draft positions can consume computation without producing much accepted output.

The published comparison is refreshingly conditional. At 60 concurrent sequences per rank, dynamic verification is estimated at 14,466 tokens per second per rank versus 13,730 for fixed-five verification, a 5.4% gain. At 16 sequences per rank, the table shows a 1.5% loss. At 96 it shows an 11.0% gain. These are DeepSeek’s estimated throughput figures, not Kingy measurements.

That is a stronger engineering lesson than a single speedup multiplier. A scheduling optimization can help a busy service while making a lightly loaded configuration slower. Test the concurrency your application actually reaches.

SWA bounded replay avoids rebuilding unnecessary decoder state

Sliding-window attention limits the local history used by a layer. DeepSeek uses that property to reconstruct a short recent suffix instead of replaying a long prompt through every decoder layer.

Its deployment notes describe reconstructing an approximate SWA cache from the latest 128 tokens. For decoder prefill, DeepSeek says this replaces exact reconstruction involving the last 2,560 tokens through 20 decoder layers, nearly halving total prefill computation in its architecture. This is a vendor claim about a particular mechanism and configuration.

The public engine’s recommended compact setting keeps 140 tokens to cover the window and backtracking allowance. That does not mean the model has only 140 tokens of total context: global/compressed context and the replayed local state have different roles.

There is also a useful distinction between implementation equivalence and model quality. The repository’s YoCo checks compare mask and compact implementations, including log probabilities and top-K outputs on a fixed trajectory. Matching those implementations does not by itself establish equivalence between approximate replay and every possible full-replay workload. Long-context answer quality still needs application testing.

Structured outputs and repeatability address application reliability

DeepInfer supports constrained output through regular expressions, JSON schemas, and EBNF grammars. The documented workflow compiles a constraint, tracks its state during decoding, and masks illegal tokens. That can reduce failures caused by malformed output. A schema-valid answer can still contain the wrong date, amount, or action, so semantic validation remains necessary.

For reproducibility, the NVIDIA path offers batch-invariant inference and deterministic sampling. The documented deterministic configuration enables batch invariance, disables variable-length MTP, and fixes the sequence seed. Preserve those settings when comparing outputs; a production configuration optimized for speed may make different choices.

What the speed benchmarks show

DeepSeek separates prefill, which processes input, from decode, which emits output. Its main README reports prefill measurements on four SM100 GPUs and decode measurements on 64 SM100 GPUs, each within one scale-up domain. The interconnect and parallel layout are part of those results.

The prefill curve rises from roughly 60,000 non-cached input tokens per second per GPU at a 4,096-token batch setting to around 89,000 at 65,536. The corresponding prefill step latency increases from roughly 70 milliseconds to 730 milliseconds. Larger batches improve throughput with diminishing returns and a latency cost.

These are prefill-step measurements. They do not include every delay a customer experiences before receiving the first token: request admission, tokenization, queueing, cache lookup, network transfer, and other stages can contribute to time to first token. “Non-cached input” also matters because reusing a prefix changes how much new computation a request needs.

The decode curve shows roughly 325 tokens per second per user and 2,600 output tokens per second per GPU at eight requests per data-parallel rank. At 256 requests per rank, it shows roughly 103 tokens per second per user and 26,000 tokens per second per GPU. Those are approximate chart readings, not raw records. They illustrate a tradeoff: raising aggregate throughput lowers individual generation speed.

Do not combine the best endpoint from each axis into a claim of 325 tokens per second per user at 26,000 tokens per second per GPU. The chart does not show that combination.

The acceptance-rate accounting changes how to read the chart

The benchmark documentation says raw decode throughput uses full-acceptance accounting: steps per second multiplied by requests and by 1 + nextn, normalized by the configured context-parallel factor. It explicitly warns that this count applies regardless of true draft acceptance.

The plotting code then scales fixed-draft results with tau = 1 + nextn × acceptance. For the published decode chart, three draft positions and 75% acceptance produce tau = 3.25. That is the chart’s stated conversion from steps to tokens.

Consequently, these curves should not be presented as direct observation of a 75% acceptance rate across an arbitrary customer workload. They combine measured step execution with the displayed acceptance assumption. A reproduction should report both the measured step rate and real accepted-token counts. The plotting code also rejects applying this multiplier to variable-length results, where acceptance is already represented in the step behavior.

The measurement harness separates profiling from clean timing, performs warmup, and excludes intervals colliding with expert load-balancing statistics flushes. Those choices help isolate steady-state engine behavior. Service-level testing should additionally include the periodic work, traffic variation, and failures that users experience.

The inspected headline charts contain no matched vLLM, SGLang, or TensorRT-LLM baseline. They establish DeepSeek’s reported operating curve on its configuration, without establishing a cross-framework winner. DeepInfer’s README credits vLLM as its principal original foundation and SGLang and TensorRT-LLM as additional influences.

Accuracy evaluations: what the reference scores mean

DeepInfer includes a separate offline evaluation workflow. Its stated purpose is to sanity-check that a V4.1 deployment is configured correctly. The reference scores describe model performance under that protocol.

Thirty-two AIME samples do not turn this into “the model solves a problem if any of 32 attempts succeeds.” The reported Pass@1 averages correctness across samples. That distinction matters when comparing it with best-of-many or Pass@k results.

Generation writes raw outputs to disk; scoring runs separately. The default scoring path uses an LLM judge. Rule-based scoring is available for AIME and GPQA, while HLE requires the LLM judge in this harness. Complete dataset coverage is a requirement for a valid Pass@1 figure. A paid external judge may add evaluation cost.

For broader model context, the 1017 model card reports 77.0 on DeepSWE v1.1, 42.3 on Terminal-Bench 4.0, and 29.1 on ProgramBench. DeepSeek identifies Mini-SWE-V2 and Claude Code as the harnesses for the first two. Those numbers belong to the checkpoint’s reported evaluations; they cannot establish that adopting DeepInfer improves a model’s coding intelligence.

An operator should validate both layers: first confirm the checkpoint’s expected behavior, then measure whether the engine and chosen optimizations preserve acceptable results for the intended workload. Neither a fast engine run nor a high model score answers both questions.

Cost: free engine code, substantial operating choices

DeepInfer is available under Apache-2.0, with no software purchase required to download the release. Hosting costs still include accelerator time, memory, storage, interconnects, CPU servers, power, engineering, and spare capacity. The economic comparison depends on utilization and the proportion of repeated context.

The deployment Q&A makes two striking claims: a single cluster serves 155 trillion tokens per day, and overall revenue is US$38 per GPU-hour at off-peak pricing. Both are DeepSeek-reported production figures. The document does not provide enough accounting detail to reproduce either independently.

US$38 is stated revenue, not GPU rental cost, net profit, or a price available to a self-hosting customer. Likewise, 155 trillion daily tokens cannot be treated as 155 trillion newly generated tokens. The disclosed figure does not give the input/output/cache-hit breakdown needed for that interpretation.

DeepSeek API prices provide a concrete alternative

The official USD table, checked October 20, lists the following for deepseek-flash, currently served by V4.1-Flash-1017. These are hosted model API rates, separate from DeepInfer’s software license.

Peak hours are 01:00–04:00 and 06:00–10:00 UTC, Monday through Friday, excluding Chinese public holidays. Other hours are off-peak. Prices can change, and separate tools or services can add charges.

As an illustrative calculation, one million uncached input tokens plus one million output tokens costs $0.75 off-peak or $1.50 peak at those rates. Substituting cache-hit input reduces those totals to $0.603 and $1.206. This example holds token counts fixed; an actual agent may use different amounts of reasoning, repeated context, retries, and tool calls.

How to calculate self-hosting cost without fooling yourself

Use measured delivered output and all-in hourly cost:

cost per million output tokens =
  total hourly deployment cost × 1,000,000
  ÷ (delivered output tokens per second × 3,600)

For a hypothetical decode-only configuration costing $5 per GPU-hour and delivering 20,000 output tokens per second per GPU continuously, the accelerator-only arithmetic is about $0.069 per million output tokens. At 25% effective utilization, it becomes about $0.278. The $5 rate, output rate, and utilization are assumptions for explaining the calculation, not a rental quote or a DeepInfer result.

Those examples omit prefill, vision encoding, storage, networking, failures, and staff. Add their costs before comparing a self-hosted deployment with an API bill that charges for both input and output. Also check whether the requested latency permits the batch size behind the throughput assumption. A server that can achieve high throughput only at unacceptable response times does not meet the application’s requirement.

For agent workloads, cost per successfully completed task is often more useful than cost per token. Include retries and review work. A lower token rate offers little benefit if the deployment completes fewer correct tasks or needs substantially more attempts.

Energy savings remain a claim to test

Reducing repeated prefill and improving accelerator utilization can reduce energy per useful result. DeepInfer’s release does not supply a controlled, matched energy comparison sufficient to quantify that saving. A defensible study would measure wall power, facility overhead, accepted tokens, and successful tasks under matched quality and latency targets. Tokens per second alone cannot establish joules per completed task.

The production deployment is much larger than the engine

The deployment notes recommend small expert-parallel groups for prefill and wide groups of 64–128 GPUs for decode. DeepSeek describes 64 prefill GPUs plus 64 decode GPUs as its smallest efficient deployment unit under the recommended layout. This is an efficiency recommendation for that production design, not a statement that every offline experiment requires 128 GPUs.

The surrounding cache system is substantial. DeepSeek describes roughly 100 PiB of SSD storage, three-day cache retention for agent traffic, and seven days for chatbot traffic. Its suggested storage server has 20 drives of 30.73 TB each and at least 40 GB/s read and 5 GB/s write throughput. These figures describe its production design and recommendations, rather than dependencies automatically provisioned by the public repository.

Why devote so much infrastructure to old context? An agent repeatedly reuses instructions, repository content, and conversation history. A cache hit can avoid processing that prefix again. DeepSeek’s design separates storage from compute so a prefill machine can reuse a prefix without being tied to the machine that originally processed it.

The public README explicitly excludes the complete KV cache system, vision/prefill/decode disaggregation service, and load balancing across data-parallel replicas. The Q&A explains their design but says cache management and load balancing remain coupled to internal infrastructure and are not yet fully open-sourced. Internal retention policies are therefore not built-in guarantees for a fresh DeepInfer deployment.

DeepSeek also says its online system reaches roughly 80% of a perfectly balanced offline benchmark. That is a useful reminder that even the developer’s production system incurs costs around the idealized engine measurement. Its published statement that internal and public end-to-end performance are at most 5% apart is another vendor comparison; the release does not provide a complete independent parity test.

deepseek-recipe fills a different gap

The companion deepseek-recipe project converts Messages, Chat Completions, and Responses requests into a common conversation format, renders model prompts, and parses generated outputs, including streaming tool calls and reasoning. It uses Rust libraries with Python bindings.

That can help developers reproduce the model’s expected protocol and prompt handling. Its documentation explicitly leaves inference, tool execution, and HTTP transport to the caller. Installing the recipe component therefore does not create a production API service or an agent execution environment.

A practical path to a first evaluation

On supported hardware, begin with the documented source build. Pin the repository commit, initialize its submodules, and use an empty Python 3.12 environment. Keep the accelerator selection exported when building and running.

git clone --recurse-submodules https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
git checkout 774ce244b87cc0eb8d4b94ab9f5de2ab22049f01
git submodule update --init --recursive
# Activate an empty Python 3.12 environment first.
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py

Use ascend for the documented Ascend backend. The default offline example downloads the 1017 checkpoint into the Hugging Face cache. If you already have the complete checkpoint locally, DEEPINFER_HF_MODEL_DIR points the default model at it. Budget model storage and download time before starting. These commands describe the documented workflow; Kingy.ai has not executed them on accelerators.

Then evaluate in four stages:

1. Check correctness. Run the offline regression and structured-output checks. Compare fixed prompts, output validity, and relevant long-context tasks. Record the exact checkpoint revision and all engine flags.
2. Reproduce an engine curve. Use the sweep script and retain its JSONL records. Record device count, topology, context, cache state, concurrency, draft length, and actual acceptance. Keep prefill and decode measurements separate.
3. Test realistic traffic. Replay representative prompt lengths, images, tool use, and arrival rates. Measure p50/p95/p99 time to first token, inter-token latency, completed-request throughput, errors, and recovery behavior. Include periodic balancing work and cache misses.
4. Compare economics and quality. Run the same workload against the current API or another supported engine with matched quality and latency requirements. Include idle capacity, cache infrastructure, input processing, and engineering time.

The AIME, GPQA, and HLE evaluation scripts require separately obtained datasets; the harness does not download them. Follow upstream access terms and record judge configuration. Repeated runs can vary, so preserve raw generations and scoring outputs instead of retaining only a final percentage.

Who should use DeepInfer now?

Infrastructure researchers and teams already serving DeepSeek at sustained scale have a strong reason to investigate. The release exposes concrete interactions between model architecture, expert placement, speculative verification, and cache reconstruction that can inform their own systems.

A company with supported accelerators, a platform team, and requirements for private deployment can use it as the basis for a measured feasibility study. It should budget the missing service components before committing to migration. Hardware support, output quality, latency under load, and cache economics are the first questions to settle.

A small application team primarily seeking inexpensive V4.1-Flash access should compare the hosted API first. Its published rates are a practical baseline, and they avoid building a distributed inference service. Self-hosting can still make sense for control or deployment requirements, but free engine code alone does not establish a financial advantage.

DeepInfer’s release makes DeepSeek’s inference engineering more inspectable and reusable. The next evidence that would change our assessment is a matched, reproducible service comparison: the same checkpoint, hardware topology, workload, output-quality checks, latency targets, actual accepted tokens, and full operating costs. Until then, use the published curves to design an evaluation and use your own completed-task measurements to decide whether to deploy.

The Kingy Brief

Get The Kingy Brief.

AI changes, original tests and one practical thing to try. Fridays at 09:00 Vancouver time.

Free · Double opt-in · Unsubscribe anytime

Privacy policy

Sign up for The Kingy Brief on Beehiiv.

Signup help and newsletter schedule

Signup form provided by Beehiiv. After submitting, check your inbox for "Confirm your subscription to The Kingy Brief" and open its confirmation link. Check Spam or Promotions if you cannot find it.

Fridays at 09:00 Vancouver time: source-checked AI changes, original tests and one practical thing to try. We skip a week when there is not enough verified material. Free. Unsubscribe anytime.

Sign up free