DeepSeek-V4-Pro

DeepSeek

Parameters

1.6T total / 49.0B active

MoE: total / active

Architecture

MoE

Released

26.06.2026

License

MIT License

Open Weights Commercial Use Multimodal BF16, I64, F32, F8_E4M3, I8 DeepSeek en zh

Input Modalities

text

Output Modalities

text

Context (native)

1,000,000 tokens

Context (extended)

1,000,000 tokens

Openness Index Score 100.0/100

About

DeepSeek-V4-Pro (deepseek-ai/DeepSeek-V4-Pro, released June 26, 2026 under the MIT License) is the preview flagship of the DeepSeek-V4 series - a 1.6T-parameter (49B activated) Mixture-of-Experts model with a one-million-token context, presented alongside DeepSeek-V4-Flash (284B, 13B activated). DeepSeek-V4-Pro-Max, its maximum reasoning effort mode, significantly advances the knowledge capabilities of open-source models - top-tier in coding benchmarks and significantly bridging the gap with leading closed-source models on reasoning and agentic tasks.

Architecture and optimization upgrades: a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) dramatically improves long-context efficiency - at 1M-token context, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2; Manifold-Constrained Hyper-Connections (mHC) strengthen residual connections for stable cross-layer signal propagation; the Muon optimizer provides faster convergence and training stability. Both models are pre-trained on more than 32T diverse, high-quality tokens, followed by a two-stage post-training paradigm: independent cultivation of domain-specific experts (SFT and RL with GRPO), then unified model consolidation via on-policy distillation into a single model. Technical report: arXiv 2606.19348.

Training Data Preview version of DeepSeek-V4-Pro. Pre-trained on 32T+ diverse tokens. Post-training: two-stage paradigm with independent domain-specific expert cultivation (SFT + RL with GRPO), then unified model consolidation via on-policy distillation. Supports three reasoning modes: Non-think, Think High, Think Max.

Benchmark Scores

Benchmark Score Date Standing
Terminal Bench 2.1
coding_agent
79.49%
13.08.2026
#22/44
better than 51%
NL2Repo
coding_agent
43.38%
13.08.2026
#26/41
better than 38%
Cybergym
general_agent
28.34%
13.08.2026
#15/17
better than 12%
DeepSWE
coding_agent
17.61%
13.08.2026
#17/21
better than 20%
DSBench-FullStack
coding_agent
11.94%
13.08.2026
#7/8
better than 14%
DSBench-Hard
coding_agent
11.55%
13.08.2026
#8/9
better than 12%
HLE (with tools)
stem_reasoning
54.90%
26.06.2026
#36/49
better than 27%
Toolathlon Verified
general_agent
47.90%
26.06.2026
#21/35
better than 41%
MMLU-Redux
knowledge
79.83%
26.06.2026
#16/30
better than 48%
MMLU-Pro
knowledge
43.55%
26.06.2026
#36/48
better than 26%

Showing 10 of 32 results

What is a good Terminal Bench 2.1 score? What is a good NL2Repo score? What is a good Cybergym score?

Hardware Requirements

Estimated VRAM per quantization at 4,096 tokens context — MoE: all experts must fit in VRAM, only 49.0B are active per token

Quantization Weights KV Cache Total VRAM On your hardware
FP16 — 16-bit 3200.0 GB 7.6 GB 3367.6 GB Which GPUs can run it? Calculate on your GPU
INT8 — 8-bit 1600.0 GB 3.8 GB 1683.8 GB Which GPUs can run it? Calculate on your GPU
INT4 — 4-bit 800.0 GB 1.9 GB 841.9 GB Which GPUs can run it? Calculate on your GPU
FP4 — 4-bit 800.0 GB 1.9 GB 841.9 GB Which GPUs can run it? Calculate on your GPU

Only datacenter GPUs — FP16 needs 3367.6 GB. For consumer hardware, choose the INT4 build with a multi-GPU setup.

Architecture

Decoder Block input Embedding vocab 129K · d 7168 Full Attention Sparse Attn 128:1 · dₕ 512 · win 128 ×61 MoE FFN 384 experts · top-6 · +1 shared · dᴻ 3072 MTP Head ×1 speculative layer Final Norm LM Head vocab 129K output
Attention
Sparse Attention (128:1)
MoE
384 experts · top-6 per token
Layers
61
Hidden size
7168
Context
1M tokens
RoPE θ
10K
Parameters
1600000M
Active params
49000M

Source: Hugging Face config.json · DeepseekV4ForCausalLM · model repo

Type: MoE
Attention: Hybrid Attention (Compressed Sparse Attention + Heavily Compressed Attention)
Decoder: Autoregressive
MoE: yes (? experts)
Routing: Expert routing

Training Pipeline

  1. 1
    pretraining

    Pre-training on 32T+ tokens

    Pre-trained on more than 32T diverse and high-quality tokens. Uses Muon optimizer for faster convergence and greater training stability. MoE architecture with 1.6T total params (49B activated).

  2. 2
    sft

    Domain-specific expert SFT

    Independent cultivation of domain-specific experts through SFT. Part of first stage of two-stage post-training paradigm.

  3. 3
    rl

    Domain-specific expert RL (GRPO)

    Independent cultivation of domain-specific experts through RL with GRPO. Part of first stage of two-stage post-training paradigm.

  4. 4
    other

    On-policy distillation consolidation

    Unified model consolidation via on-policy distillation, integrating distinct proficiencies across diverse domains into a single model. Second stage of two-stage post-training paradigm.

Linked Resources

arxiv

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

https://arxiv.org/abs/2606.19348

 website

DeepSeek Homepage

https://www.deepseek.com/

 demo

DeepSeek Chat

https://chat.deepseek.com/

 citation

DeepSeek-V4 Citation

https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro

Comments (0)

No comments yet

Be the first to share your thoughts!

What now?

You have read the full card for DeepSeek-V4-Pro — here is where to go next.

Compare with another model Test on your own GPU Related models Data as JSON