repowisedeepseek-ai/deepinfer

deepseek-ai / deepinfer

DeepInfer: High-performance, large-scale inference framework for DeepSeek models

Python★ 21,528
Use in your agentAsk this codebase

github.com/deepseek-ai/deepinferIndexed at 9bd9cf2Up to date with upstream

What is deepinfer?

deepinfer is a Python inference framework for offline, large-scale inference with DeepSeek models. It consumes model checkpoints, builds or loads an execution plan, and produces inference outputs through scheduling, post-processing, and execution components. The project uses Python with C++ and CUDA back-end code and supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Written by a model from the README and the source

326 files and 2,950 symbols in 8 modules, led by Python (274 files) and C++ (33).

Start reading

Code health

8.1out of 10Good
Full health report

On the leaderboards:#5 of 50 Python repos

Use deepseek-ai/deepinfer in your agent

Point Claude Code, Cursor, Codex or VS Code at this repository. Your agent gets the index this page is built from: cited answers, callers, history and health for any file.

  1. 1.In Claude, open Customize → Connectors, click +, then Add custom connector. On a Team or Enterprise plan an owner adds it first under Organization settings → Connectors.
  2. 2.Paste the server URL below as the Remote MCP server URL, then click Add.
  3. 3.Click Connect. A Repowise sign-in window opens. Approve access.
  4. 4.Repowise's tools are now available in your Claude chats.

Connect with one click. Sign in with Repowise (OAuth), no API key to copy. Works in Claude on the web, desktop, and mobile.

Docs
bash
https://api.repowise.dev/mcp/deepseek-ai/deepinfer
Editors need an API key: sign in to create one.Sign in to connect

This repo is public, so anyone with a Repowise account and an API key can connect.

Try asking

  • get_overview()
  • get_answer("how is deepinfer put together?")
  • get_why("why is deepinfer structured this way?")

Explore deepinfer

Key features of deepinfer

  • DeepInfer supports optimized kernels, model runtime, and sampler runtime features.
  • DeepInfer supports Attention DP, balanced CP, MoE EP, and EPLB parallelism.
  • DeepInfer supports DSpark and confidence-scheduled verification for decoding.
  • DeepInfer provides asynchronous scheduling that overlaps CPU batch scheduling with device execution.
  • DeepInfer supports batch-invariant inference and deterministic sampling under the documented settings.
  • DeepInfer supports structured outputs constrained by a regex, JSON schema, or EBNF grammar.

How deepseek-ai/deepinfer works

Input begins with a model checkpoint downloaded from Hugging Face by default or supplied through DEEPINFER_HF_MODEL_DIR. The deepinfer package builds or loads an execution plan and coordinates model and sampler runtime behavior. The csrc/engine module prepares scheduling state, FLOPs estimates, and sample input shapes for the engine. The csrc module provides low-level timing and device-memory helpers exposed to Python through pybind, including CUDA- and NPU-specific memory-sharing utilities. The deep_cluster module selects an accelerator backend and aligns process CPU and memory placement with device NUMA locality. The eval module formats benchmark inputs and scores generated outputs.

The codebase has 326 files and 2,950 symbols in 8 modules, led by Python, C++ and Shell. Code health is 8.1 out of 10, rated good.

Getting started with deepinfer

  1. Create and activate an empty Python 3.12 environment, then select the accelerator and build DeepInfer from source.
    export DEEPINFER_ACCELERATOR=nvidia
  2. Run the offline inference example.
    python examples/offline_inference.py

Commands copied from the README

deepseek-ai/deepinfer FAQ

What is deepinfer?

deepinfer is an inference framework for offline, large-scale inference with DeepSeek models. It provides runtime, scheduling, and execution components for supported hardware.

How does deepinfer work?

deepinfer consumes a model checkpoint, builds or loads an execution plan, and runs inference through its Python engine and C++/CUDA back end. It also handles scheduling, post-processing, and evaluation.

How do I install deepinfer?

deepinfer is built from source in an empty Python 3.12 environment. Set DEEPINFER_ACCELERATOR and run ./build.sh.

What language is deepinfer written in?

deepinfer is primarily written in Python, with C++ and CUDA back-end code. Its dependencies include pybind11, numpy, and huggingface_hub.

How is the deepinfer code organized?

deepinfer contains the main Python package, C++ sources in csrc, accelerator placement code in deep_cluster, evaluation code in eval, examples, tests, scripts, and packaging configuration.

When should I use deepinfer?

Use deepinfer for offline inference with supported DeepSeek models on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs. The project provides features for serving inference at scale.

What license does deepinfer use?

deepinfer uses the Apache-2.0 license.

The summary, features and answers on this page are written by a model from the README and the source, and refreshed with each index.

About this page

Get a page like this for your own codebase

Repowise built this map of deepseek-ai/deepinfer from its source and git history. It parses every symbol, draws the dependency graph, scores each file's health from complexity, duplication, coverage and churn, finds hotspots and owners in the history, and writes documentation for every module. Your coding agent reads the same index over MCP, so its answers cite real files.

Index your repo freeSee how Repowise worksFree for 2 public repos, with a page like this one.

Prefer local? pip install repowise && repowise init indexes your own checkout, no account needed.