DeepSeek DeepInfer Guide: Requirements, Offline Inference, and Benchmark Checks
DeepInfer is the inference engine DeepSeek says powers its large-scale production services, now released under Apache 2.0. It is not a new model trainer or a consumer desktop runner. Its job is to distribute DeepSeek-V4.1-Flash inference across accelerators, schedule prefill and decode work efficiently, and make outputs easier to validate and reproduce.
That distinction matters before you clone the repository. The current release officially supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It focuses on offline inference and does not include every component required for a complete online service, including a KV-cache system, full VE/prefill/decode disaggregation, and load balancing across data-parallel replicas.
This guide therefore uses a safer goal: decide whether your infrastructure qualifies, build the official project in a clean environment, run the offline path, and then validate deterministic output, structured output, and performance tradeoffs without treating DeepSeek's published benchmark as a promise for different hardware.
Who can realistically use DeepInfer today?
| Environment | Recommendation | Why |
|---|---|---|
| Laptop or consumer GPU | Not a practical target yet | It is outside the official hardware list, and the model and parallel design assume large infrastructure. |
| SM100 or Ascend 950 cluster | Suitable for evaluation | The repository documents build, offline inference, benchmark, and evaluation paths. |
| Team operating vLLM or SGLang | Worth a controlled comparison | DeepInfer started primarily from vLLM and integrates DeepSeek's kernel and parallelism projects. |
| Team needing a turnkey API server | Plan additional engineering | The open release does not include several production serving layers. |
What DeepInfer changes in the inference stack
Prefill and decode are treated as different workloads
Large-model inference has two distinct phases. Prefill processes the input tokens, while decode generates subsequent tokens step by step. Their useful metrics are not identical. DeepInfer's benchmark separates prefill batch size and non-cached input work from the decode request batch. Prefill reporting emphasizes input throughput per accelerator and latency. Decode reporting combines steps per second, output throughput per accelerator, and the speed seen by each request.
This is a more useful way to make a deployment decision than chasing one maximum number. An interactive assistant may prioritize per-user token speed, while a background batch system may accept slower individual streams in exchange for higher aggregate throughput.
It combines DeepSeek's kernels and parallel strategies
The project lists Attention DP, balanced context parallelism, MoE expert parallelism, and expert-parallel load balancing. It also integrates FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. This is why DeepInfer should be evaluated as cluster infrastructure rather than a one-click local application.
Reproducibility and constrained output are first-class features
Batch-invariant inference is designed to keep model results stable when batch composition changes. Deterministic sampling can reproduce output for the same input tokens, sampling parameters, and sequence seed. Structured output can constrain generation with a regular expression, JSON Schema, or EBNF grammar.
These controls are valuable when an agent must produce API arguments, workflow records, or evaluation artifacts that downstream software will parse. On the NVIDIA path, the documented deterministic setup enables batch invariance, disables variable-length MTP, and passes a fixed seq_seed to SamplingParams. Do not describe a run as deterministic unless the relevant settings and inputs are captured together.
Prerequisites to check before installation
- Operating system: Linux
- Compiler: GCC 13 or newer
- Python: a clean Python 3.12 environment
- NVIDIA path: SM100 GPUs with CUDA 13
- Ascend path: Ascend 950 NPUs with CANN 9.2
- Model storage: enough space for DeepSeek-V4.1-Flash-1017 in the Hugging Face cache, or an existing local checkpoint directory
The accelerator architecture is the critical gate. Having CUDA installed is not sufficient if the GPU is outside the supported architecture. DeepSeek also states that its published prefill plot was measured on four SM100 GPUs and its decode plot on 64 SM100 GPUs, each within one scale-up domain. Those curves should not be converted into an estimate for a small or different cluster.
Build DeepInfer and run the first offline inference
Step 1: Clone the repository and create a clean Python environment
git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
python3.12 -m venv .venv
source .venv/bin/activateConfirm that python --version reports Python 3.12 and that the repository root contains build.sh and the examples directory. A clean environment makes compiler and backend dependency failures easier to isolate.
Step 2: Select the accelerator backend and build
# Use ascend instead of nvidia for the Ascend path
export DEEPINFER_ACCELERATOR=nvidia
./build.shKeep DEEPINFER_ACCELERATOR exported in the shell where you run DeepInfer, not only while building. The build script installs backend-specific requirements, vendored operators, and the in-tree DeepInfer extension. If compilation fails, verify the supported device architecture, GCC, Python, and CUDA or CANN versions before changing package pins or compiler flags.
Step 3: Run the official offline example
python examples/offline_inference.pyWhen the default model ID is used, the first run downloads DeepSeek-V4.1-Flash-1017 to the Hugging Face cache. A restricted environment can point the default model to a local directory:
export DEEPINFER_HF_MODEL_DIR=/models/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.pyJudge completion by more than process startup. Check that sample generation finishes and that the documented decode_ppl comparison completes. The official example also supports --check_structured_output, which reuses the same model instance to test regex, JSON, and EBNF constraints and verifies normal EOS termination.
Three practical validation workflows
1. Use offline inference as an infrastructure smoke test
The first goal is not to claim a performance win. It is to confirm that the engine, checkpoint, operators, and devices work together. Record the selected backend, model location, visible devices, parallel settings, build commit, and whether sample generation and PPL validation complete. If the run fails, fix dependency and device issues before comparing model quality.
2. Regression-test JSON output for an agent
Suppose an agent must return tool arguments as JSON. Apply a JSON Schema constraint, use fixed prompts and sampling settings, and run the same cases under multiple valid batch compositions. On NVIDIA, align the batch-invariant and deterministic-sampling settings and use a fixed sequence seed.
The review criteria should include successful parsing, required-field presence, absence of out-of-schema fields, normal EOS termination, and equality across the intended repeated runs. A parseable object alone is not enough if values drift in a workflow that requires reproducibility.
3. Find the right throughput and latency operating point
examples/benchmark.py lets the operator vary decode request count, context length, prefill batch size, and other settings. Larger decode batches can raise total output throughput while reducing the token rate experienced by each request. Run a sweep for the workload you actually serve, save the full configuration next to the metrics, and select an operating point based on the service-level target rather than the highest headline throughput.
The official benchmark documentation warns that profiler sections add immediate overhead and that step times inside those traces should be used to inspect kernel composition, not to derive steady-state steps per second. Keep profiling and production-style measurement runs separate.
How to read the official benchmark responsibly
The point labels in the decode chart represent requests per data-parallel rank. The documented curve uses a 131,072-token context and a particular combination of experts, next-token drafting, and assumed acceptance. Changing the model, accelerator count, context length, expert layout, or network topology changes the result.
The reported decode throughput also follows the benchmark's full-acceptance accounting formula. It is not a direct measurement of the acceptance rate for every workload. For a reproducible internal comparison, preserve the JSONL result, software commit, environment variables, device topology, and input profile. Measure power, cost, queue time, and tail latency separately if they matter to the deployment decision.
Evaluation is separate from serving performance
The repository includes offline batch evaluation for AIME 2026, Humanity's Last Exam, and GPQA Diamond. Generation and scoring are separate stages. Raw outputs can therefore be rescored with a different judge without regenerating them.
The scripts do not automatically download the datasets. Operators must obtain them from their upstream sources and convert them to the documented JSONL structure. HLE and some GPQA sources can require accepting dataset terms. HLE supports the LLM judge path because much of its answer set is free-form, while AIME and GPQA also provide deterministic rule-based scoring.
Before reporting Pass@1, inspect summary.json. The result is only complete when coverage_complete is true; otherwise one or more ranks or samples are missing. Model-quality numbers and serving-throughput numbers answer different questions and should not be combined into a single claim of superiority.
What is still needed for an online service
- A KV-cache storage and retrieval layer
- VE, prefill, and decode disaggregation with failure handling
- Load balancing across prefill and decode data-parallel replicas
- Admission control, queue limits, timeouts, and observability
- Secure checkpoint distribution and cache permissions
- Prompt, output, and evaluation-data handling rules
DeepInfer's code is Apache 2.0, but the DeepSeek-V4.1-Flash-1017 model has its own model license. Evaluation datasets also have separate terms. Treat code, checkpoint, and data licensing as three different checks.
Troubleshooting in the right order
Operator compilation fails
Verify a clean Python 3.12 environment, GCC 13 or newer, and the required CUDA 13 or CANN 9.2 stack. Then confirm the device is actually SM100 or Ascend 950 and that DEEPINFER_ACCELERATOR matches the platform.
The first run stalls while loading the model
Check network policy and Hugging Face cache capacity. If the checkpoint was transferred separately, set DEEPINFER_HF_MODEL_DIR to the local directory. An explicitly passed --model_name takes precedence over that environment variable.
Your benchmark is far from the published curve
Compare accelerator count, EP/DP/CP layout, context length, non-cached tokens, nextn, redundant expert count, and visible-device settings. In multi-node runs, also verify WORLD_SIZE and RANK. A different hardware scale should be expected to produce a different operating curve.
Evaluation output is incomplete or unexpectedly low
Validate the JSONL fields, image placeholders, judge configuration, and the existence of raw output from every rank. Finish by checking coverage_complete rather than relying only on the displayed score.
Bottom line
DeepInfer is important because it exposes the kernels, parallel execution, scheduling, and reproducibility mechanisms behind DeepSeek's large-scale inference stack. It does not make that stack a turnkey local runner. The practical adoption sequence is to confirm supported hardware, build in a clean Python 3.12 environment, validate offline and structured outputs, reproduce a workload-specific Pareto curve, and design the missing serving layers before production use.
Official sources
- DeepSeek DeepInfer repository and README
- Official examples and benchmark documentation
- Official evaluation documentation
- DeepSeek-V4.1-Flash-1017 model card
Sources checked October 22, 2026.
Comments
Post a Comment