DeepSeek has released DeepInfer, the inference framework that it says runs its production service for DeepSeek-V4.1-Flash. The code is on GitHub under the Apache 2.0 license. In the launch post on X on October 20, DeepSeek said it shares the stack "to help the community lower compute costs, reduce energy use, and make AI more affordable."
The release also put two business numbers on the record. In its deployment Q&A, DeepSeek says a single cluster serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour at valley (off-peak) pricing.
This post explains what DeepInfer includes, what it leaves out, and what the numbers do and do not show.
What DeepInfer is
DeepInfer is the software layer between a trained model and the people who send it requests. It schedules requests onto GPUs, runs the model kernels, decodes tokens, and returns output. The README says the open-source release provides offline inference "with the core features needed for serving at scale."
Facts from the repository:
- Supported model: DeepSeek-V4.1-Flash only. The default checkpoint is DeepSeek-V4.1-Flash-1017, whose weights are MIT-licensed.
- Supported hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
- Requirements: Linux, GCC 13 or later, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend).
- Kernels: DeepSeek's own FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect libraries, plus custom TileLang kernels.
- Origins: the README says DeepInfer "was originally built primarily on vLLM," with ideas from SGLang and TensorRT-LLM.
DeepSeek also released deepseek-recipe, an MIT-licensed component that converts Chat Completions, Responses and Messages API requests into model token sequences. DeepSeek says its logic is "fully aligned with the official DeepSeek API," so a third-party host can give the same API behavior as DeepSeek's own service.
On X, DeepSeek said inference code for future DeepSeek models will also land in the same repository.
What is not in the release
The README lists three parts that a complete online service needs and that this release does not include:
- A KV cache system (the store that keeps earlier computation so the model does not redo it)
- Disaggregation of the vision encoder, prefill and decode stages onto separate machines, though DeepSeek says "most required engine functionality is already implemented"
- Load balancing across data-parallel replicas for prefill and decode
The Q&A says the KV cache and load balancing code are "tightly coupled to our internal infrastructure" and are "not yet fully open-sourced." It describes their design in prose instead, with a longer write-up in a Zhihu post.
So DeepInfer is not a drop-in replacement for DeepSeek's API. It is the engine, and an operator must build the cache and routing layers around it.
Main techniques
The Q&A gives more detail than most lab releases. The main points:
- Disaggregated serving. Prefill (reading the prompt) and decode (writing the answer) run on different machines. DeepSeek uses ordinary 8-GPU machines for prefill and 64 to 128-GPU supernodes for decode.
- Wide expert parallelism. Decode spreads the mixture-of-experts layers across 64 to 128 GPUs. The Q&A says the smallest efficient unit is 64 GPUs of prefill plus 64 GPUs of decode. That puts full production-style serving out of reach for small teams.
- Long-lived KV cache. The design goal is to keep KV cache for more than 72 hours. The production cache sits on DeepSeek's 3FS file system, with about 100 PiB of SSD storage, a 3-day TTL for agent traffic and a 7-day TTL for chatbot traffic.
- DSpark speculative decoding. A draft model writes 5 tokens at a time, and a confidence head decides which draft tokens are worth checking. DeepSeek reports +4 to 7% throughput at its production operating point versus always checking all 5.
- SWA bounded replay. When part of the cache is missing, the engine recomputes only the most recent 128 tokens instead of a long window. The Q&A says this nearly halves total prefill compute in one path.
- Deterministic output. Batch-invariant inference and deterministic sampling give identical results for identical inputs, on NVIDIA only.
- Low precision. Routed-expert weights and part of the KV cache use 4-bit formats (MXFP4 and an NVFP4-like format).
DeepSeek also says its internal FlashMLA has one hand-tuned optimization that the public version lacks, which makes the decode kernel 5 to 10% slower. It puts the end-to-end gap at no more than 5%.
What the $38 figure means
The 155 trillion tokens per day and $38 per GPU-hour figures are DeepSeek's own claims about its production service. They are not benchmarks of the open-source code, and DeepSeek does not say how many GPUs the cluster has or which chips it uses. Revenue per GPU-hour is also not profit: it leaves out the cost of hardware, power, storage and staff.
Still, the number gives outsiders a reference point for one model's serving economics. DeepSeek's prices are among the lowest in the market, as our LLM API pricing comparison shows. DeepSeek's Q&A says the aim is to serve more requests with less compute and energy, "so that intelligent services reach more users at a lower price." Its Q&A describes a KV cache kept for days, built so that repeat context does not have to be computed again.
Why it matters
Three reasons stand out.
First, Ascend support. DeepInfer runs on Huawei Ascend 950 NPUs as well as NVIDIA GPUs. The Q&A names "support accelerators from multiple vendors" as the first design goal of the system. Operators without SM100-class NVIDIA GPUs get a second supported platform for DeepSeek-V4.1-Flash.
Second, third-party hosting. Inference providers that serve DeepSeek's open weights can now copy DeepSeek's own engine and API behavior, instead of reverse-engineering it. DeepSeek says deepseek-recipe gives third-party deployments "the same API behavior as the official one."
Third, policy. The release comes the same week the US Department of Homeland Security ordered DeepSeek products removed from federal civilian agency systems, an order that also covers third-party products developed using DeepSeek.
Who should look at it
- Teams that already serve DeepSeek-V4.1-Flash-1017 on SM100-class GPUs or Ascend 950, and want DeepSeek's reference engine
- Inference engineers who want detailed, production-tested choices on prefill scheduling, cache design and speculative decoding
- Not hobbyists on consumer GPUs: the supported hardware is datacenter-class, and DeepSeek's efficient decode setup starts at 64 GPUs
