DeepSeek Open-Sources DeepInfer, Its In-House AI Inference Framework

DeepSeek has released DeepInfer, an in-house artificial intelligence inference framework designed to run its large language models more efficiently at scale. Announced on October 20, 2026, the framework powers the company’s large-scale production services for DeepSeek V4.1 Flash and is now available as open-source software under the Apache 2.0 licence.

The release gives developers and businesses access to infrastructure optimizations that DeepSeek says it uses to serve AI requests with high throughput and low latency. DeepInfer includes optimized computational kernels, distributed inference, speculative decoding, asynchronous scheduling, structured output generation and other features designed to improve the efficiency of large-scale AI serving.

The framework currently supports DeepSeek V4.1 Flash and targets NVIDIA SM100 GPUs and Huawei Ascend 950 neural processing units (NPUs). DeepSeek has also acknowledged important limitations: the public release does not include every component needed to reproduce its complete production serving infrastructure, particularly its full KV-cache management and load-balancing systems.

The move is significant because AI inference—the process of generating responses after a model has been trained—has become a major operational expense for AI companies. By open-sourcing the software behind its serving infrastructure, DeepSeek is giving developers a closer look at how it optimizes large-scale AI workloads while potentially helping organizations already running its models reduce infrastructure costs.

Key takeaways

• Open-source release: DeepSeek has made DeepInfer available under the Apache 2.0 licence.
• Production technology: The framework powers large-scale inference services for DeepSeek V4.1 Flash.
• Supported hardware: It currently targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
• Performance optimization: Features include optimized kernels, distributed inference, speculative decoding and asynchronous scheduling.
• Large-scale serving: DeepSeek says its production cluster serves approximately 155 trillion tokens per day.
• Not a complete deployment stack: Some production infrastructure, including the full KV-cache management and load-balancing components, remains outside the open-source release.
• Developer access: The code is available through the official DeepSeek-AI/DeepInfer GitHub repository.

What is DeepSeek DeepInfer?

DeepInfer is an inference framework developed by DeepSeek to run large language models efficiently in production environments.

An AI model’s performance depends on more than its underlying architecture or the quality of its training. Once deployed, the model must process incoming requests, manage memory, distribute computation across accelerators and generate output tokens while balancing speed, cost and reliability.

These operations can become especially demanding when a service handles millions of conversations or agent tasks simultaneously.

DeepInfer addresses this infrastructure challenge. It brings together optimized computational kernels and an inference engine designed to coordinate model execution across supported hardware.

According to DeepSeek’s official repository, the framework provides the core capabilities required for offline inference and includes features developed for large-scale model serving.

The software is not another chatbot or a newly trained language model. Instead, it is the infrastructure that helps an existing model execute efficiently.

For developers, this distinction matters. A language model determines what an AI system can generate, while an inference framework influences how efficiently that capability can be delivered to users.

Why has DeepSeek released its in-house inference framework?

DeepSeek’s release comes as AI companies increasingly focus on inference efficiency rather than model quality alone.

Training a large model can require substantial computing resources, but running it for millions of users creates a continuing operational expense. Every request consumes computing time, memory bandwidth and other infrastructure resources.

For applications that generate long responses or execute multi-step AI-agent workflows, these costs can accumulate quickly.

DeepSeek says its inference system was designed around three broad objectives:

1. Support accelerators from different hardware vendors.
2. Optimize the distribution of prefill and decoding workloads across available computing resources.
3. Retain key-value cache information for longer periods to reduce repeated computation when requests share reusable context.

These objectives address common bottlenecks in large-scale AI deployment.

By releasing DeepInfer, DeepSeek allows developers to examine and reuse many of its implementation techniques rather than relying exclusively on the company’s hosted services.

The release could also encourage outside developers to identify bugs, contribute improvements and adapt the framework to supported deployment environments.

However, open-sourcing an inference framework does not automatically make the underlying model free of all usage costs. Organizations still need compatible hardware, computing capacity, engineering resources and the model weights required for their intended deployment.

DeepInfer’s main features explained

DeepInfer combines several techniques that address different parts of the inference process.

Feature	What it does	Why it matters
Optimized kernels	Uses specialized implementations for computational operations	Can reduce execution overhead
Distributed inference	Coordinates work across multiple accelerators	Enables large models to run across hardware clusters
Speculative decoding	Uses draft tokens and verification to accelerate generation	Can improve decoding throughput
Asynchronous scheduling	Overlaps CPU scheduling work with accelerator execution	Reduces idle time
Expert parallelism	Distributes mixture-of-experts computation across devices	Helps scale model execution
Expert-parallel load balancing	Redistributes expert workloads to reduce uneven utilization	Can prevent some devices from becoming bottlenecks
Structured outputs	Constrains generation to formats such as JSON schemas or grammars	Makes model responses easier to integrate into applications
Deterministic sampling options	Supports repeatable generation under specified configurations	Helps with testing and reproducibility

These capabilities are intended to work together. Faster individual kernels alone may not deliver the best end-to-end results if scheduling, communication or memory management becomes the bottleneck.

DeepInfer therefore combines low-level computation optimizations with higher-level techniques for coordinating inference workloads.

1. Optimized kernels improve how AI models execute

A computational kernel is a piece of software that performs a particular operation, such as matrix multiplication or an attention-related calculation.

Large language models repeatedly execute these operations while processing input tokens and generating responses. Even small inefficiencies can become significant when the same operations run billions of times across a production workload.

DeepInfer incorporates optimized kernels and related libraries, including FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, alongside custom TileLang kernels.

These components address different computational tasks, from matrix operations to attention and communication-heavy parts of model execution.

The purpose is to make better use of the available accelerator hardware and reduce unnecessary overhead.

For an organization serving many concurrent requests, such improvements may translate into higher throughput or lower latency. The actual benefit depends on the model, hardware configuration, workload and software environment.

DeepSeek has not published a single universal percentage improvement in the repository that applies to every deployment. Performance claims should therefore be evaluated using benchmarks that reflect the intended production configuration.

2. Speculative decoding can accelerate token generation

One of DeepInfer’s important features is speculative decoding, including a technique DeepSeek calls DSpark.

Large language models normally generate text sequentially, producing one token after another during decoding. This process can be expensive because each generation step requires model computation.

Speculative decoding attempts to reduce this cost by using a faster draft process to propose several tokens. A larger or more expensive verification process then checks the proposed tokens and accepts those that meet the model’s requirements.

If several tokens can be accepted together, the system may generate output more efficiently than through ordinary sequential decoding.

DeepInfer also supports confidence-scheduled verification. Rather than treating every draft token identically, the system uses confidence estimates to help determine how much verification work should be allocated to different proposals.

This matters because the probability of accepting a draft token can vary across positions and requests. Verifying low-confidence tokens may consume resources without delivering equivalent throughput gains.

By adjusting the verification strategy, the framework aims to use computing resources more effectively.

Speculative decoding does not guarantee the same speedup for every model or workload. The benefit depends on factors such as draft quality, acceptance rates, hardware performance and the amount of concurrent work.

3. Asynchronous scheduling reduces idle time

Another feature is asynchronous scheduling, which coordinates CPU-side scheduling with accelerator execution.

During inference, the CPU performs tasks such as deciding which requests should run next, preparing work and processing intermediate results. The accelerator executes the model’s computational operations.

If these activities are tightly synchronized, the GPU may wait for CPU work to finish before continuing. Even a powerful accelerator can lose efficiency when it repeatedly sits idle.

DeepInfer’s asynchronous scheduler aims to overlap scheduling and execution so that CPU work can proceed while the GPU processes an earlier batch.

This approach is particularly relevant to high-throughput deployments in which many requests are processed concurrently.

Reducing idle time can improve the number of tokens generated over a given period without necessarily changing the underlying model.

The effectiveness of the approach still depends on workload characteristics, hardware communication and the balance between CPU and accelerator resources.

4. Distributed inference helps DeepSeek use multiple accelerators

Large language models can exceed the memory capacity or practical computing limits of a single accelerator. Distributed inference allows the system to divide computation across multiple devices.

DeepInfer supports several forms of parallelism, including attention data parallelism, context-related parallelism and expert parallelism for mixture-of-experts models.

Mixture-of-experts, or MoE, architectures contain multiple specialized neural-network components called experts. Depending on the model architecture, only a subset of these experts may be activated for a particular token.

Expert parallelism distributes those components across devices. The system must then coordinate communication and computation so that workloads are processed efficiently.

This creates a balancing challenge: if some devices receive substantially more work than others, the entire operation may slow down while the busiest devices finish.

DeepInfer includes expert-parallel load balancing (EPLB) mechanisms intended to reduce this imbalance by tracking expert workloads and adjusting their placement.

These techniques become increasingly important as inference moves from single-machine deployments to large accelerator clusters.

5. Structured outputs make AI responses easier to use

DeepInfer also supports structured output generation.

Many business applications require machine-readable responses rather than free-form prose. An application might need the model to return a valid JSON object, extract specified fields or follow a particular grammar.

Without constraints, a model can generate a response that contains the right information but fails to follow the format expected by the application.

DeepInfer can constrain output using supported formats such as JSON schemas, regular expressions and EBNF grammars.

The framework uses grammar-related processing to determine which tokens are valid at a given generation step and restricts output accordingly.

This capability can be useful for AI agents, data extraction systems and software that passes model output directly to other components.

Structured output does not guarantee that the content is factually correct. It helps ensure that the generated response follows a required format, while the application remains responsible for validating the meaning and safety of the result.

What hardware does DeepInfer support?

The public repository identifies two supported accelerator platforms:

• NVIDIA SM100 GPUs
• Huawei Ascend 950 NPUs

The NVIDIA SM100 designation refers to NVIDIA’s Blackwell Ultra-class accelerator architecture. Huawei Ascend is a family of AI accelerators developed in China.

Supporting both platforms is notable because AI infrastructure providers increasingly seek alternatives to relying on a single hardware ecosystem.

DeepSeek’s implementation includes platform-specific dependencies. The repository specifies CUDA 13 for NVIDIA deployments and CANN 9.2 for Huawei Ascend deployments, alongside Linux, GCC 13 or later and Python 3.12.

These requirements mean DeepInfer is not a general-purpose desktop application that can be installed on any computer. It is intended for specialized AI inference environments.

Businesses interested in using the framework should check the current repository instructions before investing in hardware or attempting a deployment.

DeepInfer and DeepSeek V4.1 Flash

The initial public release supports DeepSeek V4.1 Flash, the model family for which DeepSeek says the framework was developed and deployed in production.

The framework is optimized around the operational requirements of that model rather than promising universal support for every third-party language model.

This is an important distinction when comparing DeepInfer with broader inference engines. An inference framework may support numerous model architectures, but specialized software can deliver advantages when it is closely aligned with a particular model’s execution patterns.

DeepSeek says its framework combines optimized kernels with an inference engine designed for distributed execution and high-throughput serving.

The repository also describes offline inference workflows, including an example that downloads the relevant model checkpoint into the Hugging Face cache on the first run. Developers can configure a local model directory when they already have a copy of the checkpoint.

The current release’s model support and hardware requirements should be treated as the baseline. Future support for additional DeepSeek models will depend on subsequent releases and development work.

How large is DeepSeek’s production inference operation?

DeepSeek’s accompanying deployment documentation provides insight into the scale of the infrastructure behind the framework.

The company says its production system currently serves approximately 155 trillion tokens per day and reports overall revenue of US$38 per GPU-hour at valley pricing.

These are figures reported by DeepSeek itself, not independently audited measurements presented in the open-source repository. The revenue figure also refers to the company’s stated pricing context and should not be interpreted as a universal rate for AI inference.

The reported scale nevertheless illustrates why inference optimization matters.

At very high request volumes, relatively small improvements in throughput, accelerator utilization or repeated computation can have a significant operational effect.

DeepSeek’s system uses a distributed architecture that separates prefill and decoding workloads, allowing resources to be allocated according to the demands of each stage.

Prefill processes the input context, while decoding generates the output tokens. Their computational characteristics differ, so separating them can help operators optimize each stage independently.

The production system also includes a long-retention KV cache, which can reduce repeated prefill computation when reusable context is available. The full cache management and load-balancing infrastructure is not included in the public DeepInfer release.

What is KV cache, and why does it matter?

The key-value cache, commonly called the KV cache, stores intermediate attention information generated while a model processes a conversation.

When a model continues an existing conversation, it may be able to reuse cached information instead of recalculating everything from the beginning. This can reduce redundant computation and improve response efficiency.

The cache becomes especially important when requests contain long contexts, repeated prompts or shared information.

However, maintaining a large cache consumes memory and requires management systems to decide what to retain, where to store it and when to remove expired information.

DeepSeek’s deployment documentation describes a system designed to retain KV-cache information for more than 72 hours under its production architecture.

This is a design objective for its broader serving system, not a guarantee that every DeepInfer installation will retain cache entries for that duration.

The company has not open-sourced all the infrastructure needed to reproduce this capability. Its deployment Q&A explains that KV-cache management and load balancing remain tightly coupled to internal systems.

This limitation matters for organizations evaluating DeepInfer as a complete replacement for an existing inference platform.

Is DeepInfer fully open source?

DeepInfer is released under the Apache 2.0 licence, which permits use, modification and redistribution subject to the licence’s terms.

However, the open-source release is not the entirety of DeepSeek’s production inference infrastructure.

The repository describes DeepInfer as providing core features for offline inference and large-scale serving. It also identifies components required for a complete inference service that are not included in the release.

These gaps include:

• A complete KV-cache management system.
• Full vision-encoder, prefill and decoding disaggregation infrastructure.
• Load balancing across data-parallel replicas for both prefill and decoding.

DeepSeek’s deployment documentation further explains that important KV-cache management and load-balancing components remain closely integrated with its internal infrastructure.

Consequently, developers can inspect and run the released framework, but reproducing DeepSeek’s entire production serving environment requires additional engineering and infrastructure.

The distinction does not diminish the value of the release. It clarifies what organizations can obtain directly from the repository and what they must build or source separately.

How DeepInfer compares with other inference frameworks

DeepInfer enters an established ecosystem of tools designed to improve the performance of large language models.

Framework	Main role	Key consideration
DeepInfer	Inference framework developed around DeepSeek models	Public release initially focuses on DeepSeek V4.1 Flash and specified accelerators
vLLM	Open-source LLM serving and inference engine	Supports a broad range of models and deployment configurations
SGLang	Framework for efficient LLM serving and structured generation	Offers serving optimizations and support for varied workloads
NVIDIA TensorRT-LLM	Optimized inference framework for NVIDIA GPUs	Closely integrated with NVIDIA’s software and hardware ecosystem

DeepSeek acknowledges that DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. Its repository thanks the developers of these projects.

This makes the release relevant not only as a new tool but also as an example of how model developers adapt existing ideas and engineering practices to meet their own production requirements.

DeepInfer’s main appeal is its alignment with DeepSeek’s own model-serving workload. Organizations using different models or hardware configurations should compare compatibility, operational complexity and measured performance before choosing a framework.

Could DeepInfer reduce AI inference costs?

DeepInfer could help reduce serving costs by improving accelerator utilization, reducing idle time and increasing the number of tokens processed with a given amount of hardware.

If a system generates more tokens per second while maintaining acceptable latency, the cost of serving each token may fall, assuming other costs remain comparable.

However, the framework’s availability does not guarantee savings for every deployment.

Actual costs depend on the accelerator type, electricity consumption, hardware rental or ownership, engineering time, workload mix and model performance. Businesses also need to account for the cost of maintaining the surrounding serving infrastructure.

For organizations already running compatible DeepSeek models, DeepInfer may provide a starting point for testing model-specific optimizations. Teams should benchmark it against their existing stack using representative prompts, context lengths, concurrency levels and latency requirements.

The strongest evidence of savings will come from measured total cost per successful task or per generated token, rather than headline performance claims alone.

What does DeepInfer mean for China’s AI hardware ecosystem?

DeepInfer’s support for both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs is significant in the context of China’s efforts to develop domestic alternatives to foreign AI hardware.

By supporting two accelerator platforms, DeepSeek is demonstrating an effort to make its inference software work across different hardware environments.

For Huawei, software compatibility is an important part of building a usable AI-computing ecosystem. Hardware performance alone is not enough: developers also need kernels, compilers, runtime software and inference frameworks that make the hardware practical for real workloads.

However, the release does not establish that Huawei accelerators match NVIDIA hardware in every performance category. Results depend on the model, deployment configuration and benchmark being measured.

DeepSeek’s continued support for NVIDIA hardware also illustrates that high-performance AI infrastructure can involve multiple vendors rather than a simple transition from one ecosystem to another.

How can developers get started with DeepInfer?

Developers can access the source code, documentation and examples through the official DeepSeek-AI/DeepInfer repository on GitHub.

The repository currently specifies the following baseline requirements:

• Linux operating system.
• Python 3.12.
• GCC 13 or later.
• CUDA 13 for NVIDIA hardware, or CANN 9.2 for Huawei Ascend.
• A supported NVIDIA SM100 GPU or Huawei Ascend 950 NPU environment.

The documented setup process includes creating a Python environment, selecting the target accelerator and building the framework from source.

Developers can then follow the offline-inference example to run the supported DeepSeek model.

Before starting, teams should check the repository for the latest installation instructions, model requirements and supported hardware configurations. Large-scale deployment will require more work than a basic offline-inference test, especially because some production components are not included.

What happens next?

DeepSeek says future model inference code and subsequent performance improvements will continue to be released through DeepInfer.

That creates an opportunity for the framework to evolve as the company’s models and serving requirements change. External developers may also contribute improvements and explore the released implementation.

The longer-term significance will depend on how well the software performs across real workloads, how quickly compatibility expands and whether the community can build the missing infrastructure needed for complete deployments.

For now, DeepInfer offers a useful view into the engineering behind DeepSeek’s large-scale inference operations. It also gives organizations running the supported model and hardware a new option to investigate when optimizing their own AI infrastructure.

Frequently asked questions

1. What is DeepSeek DeepInfer?

DeepInfer is an open-source AI inference framework developed by DeepSeek to run its models efficiently. It includes optimized kernels, distributed inference, speculative decoding, asynchronous scheduling and structured output support.

2. When did DeepSeek release DeepInfer?

DeepSeek’s DeepInfer repository appeared publicly on October 20, 2026, alongside documentation describing the framework and its deployment architecture.

3. Is DeepInfer free to use?

The framework is released under the Apache 2.0 open-source licence. However, operating it can require compatible accelerators, model weights, infrastructure and engineering resources, which may incur costs.

4. Which AI model does DeepInfer support?

The current public repository lists DeepSeek V4.1 Flash as its supported model.

5. Which GPUs and accelerators are compatible with DeepInfer?

The repository identifies NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Developers should check the latest requirements before attempting installation.

6. Does DeepInfer include DeepSeek’s entire production infrastructure?

No. The public release does not include all components required to reproduce the full production serving system. In particular, complete KV-cache management, load balancing and some distributed deployment components remain outside the release.

7. Can DeepInfer reduce AI inference costs?

It may reduce costs by improving throughput and hardware utilization, but savings depend on the model, workload, hardware, operating costs and deployment configuration. Organizations should benchmark it before making cost assumptions.

8. How does DeepInfer accelerate AI generation?

It combines optimized computational kernels, distributed inference, speculative decoding and asynchronous scheduling. These techniques aim to reduce computation overhead, improve accelerator utilization and generate output more efficiently.

9. How can developers download DeepInfer?

The source code and documentation are available at the official DeepSeek-AI/DeepInfer GitHub repository.