DeepSeek has open-sourced DeepInfer, the high-performance inference framework it runs in production. The repository went to the top of Hacker News today with more than 1,900 points. Released under Apache 2.0, DeepInfer ships as offline inference with the core pieces needed to serve DeepSeek-V4.1-Flash at scale, giving founders a direct look at how the company pushes its own models through GPUs.
The feature list is unusually detailed for a launch release. It includes optimized kernels from DeepSeek's own stack (FlashMLA, DeepGEMM, DeepEP, DeepSelect and TileLang), attention data parallelism with balanced context parallelism, and MoE expert parallelism with EPLB. Decoding gets DSpark plus confidence-scheduled verification, and the scheduler is asynchronous so CPU batch planning overlaps device execution. There is also batch-invariant inference, deterministic sampling, structured outputs constrained to regex, JSON schema or EBNF grammar, and SWA bounded replay that trims decoder prefill to a fixed suffix. Supported hardware is Nvidia SM100 GPUs and Huawei Ascend 950 NPUs.
Founders should read the fine print before swapping out a serving stack. DeepInfer is an offline inference framework, not a complete service: the KV cache system, prefill-decode disaggregation and cross-replica load balancing are explicitly not in this release. It needs Linux, GCC 13 or newer, Python 3.12 and either CUDA 13 or CANN 9.2, and the reference benchmark numbers come from 4 SM100 GPUs for prefill and 64 for decode. The codebase started from vLLM with ideas borrowed from SGLang and TensorRT-LLM, so teams already on those engines can compare implementations directly. The Ascend 950 support is the notable signal: DeepSeek is building a serving path that runs on Chinese silicon as well as Nvidia.