DeepSeek released DeepInfer, its in-house inference framework, under the Apache 2.0 license on October 20, 2026. The framework powers DeepSeek-V4.1-Flash in production, delivering high throughput and low latency at scale. DeepSeek stated its aim as helping the community lower compute costs, reduce energy use, and make AI more affordable.
DeepInfer originated from the open-source vLLM framework but has been reconstructed bottom-up, with core scheduling logic, data preprocessing pipeline, and sampling computation modules rewritten in C++, leaving minimal residual similarity to the original vLLM architecture. Hardware support is limited to NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 chips; model support covers only DeepSeek-V4.1-Flash. The release covers offline inference only and omits production-serving components including KV cache management, prefill/decode disaggregation, and load balancing, leading developers to describe it as a reference implementation rather than a production-ready server.
Technical highlights include DSpark, a speculative decoding component that drafts up to 5 tokens ahead and uses confidence estimates to schedule verification, and a bounded replay technique that trims prefill cost by recomputing only a 140-token suffix aligned to the sliding attention window. A companion library, deepseek-recipe, converts Chat Completions, Responses, and Messages API requests into DeepSeek V4/V4.1 prompts matching official API behavior.
The repository accumulated over 6,300 GitHub stars within eight hours of publication. Interconnects.ai estimated DeepInfer to be far more efficient than existing open-source tools like vLLM and SGLang for DeepSeek's models, and said estimates point to DeepSeek achieving up to 90% gross margin selling tokens on this stack. Interconnects.ai described it as the first time a leading AI lab has shared its internal inference tooling publicly, and said wide adoption by inference companies is expected to drive down prices for DeepSeek model tokens across the ecosystem.
Separately, the U.S. Department of Homeland Security is ordering civilian federal agencies to remove DeepSeek products from their systems.