NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16

NVIDIA

Parameters

550.0B total / 55.0B active

MoE: total / active

Architecture

Mamba2-Transformer Hybrid Latent Mixture of Experts (LatentMoE) with Multi-Token Prediction (MTP)

Released

03.06.2026

License

OpenMDW License Agreement v1.1

Open Weights Commercial Use Multimodal BF16 Nemotron en fr es it de ja hi ko pt zh

Input Modalities

text

Output Modalities

text

Context (native)

262,144 tokens

Context (extended)

1,000,000 tokens

Openness Index Score 70.0/100

About

NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 (nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) is a frontier-scale LLM from NVIDIA's Nemotron 3 family (released June 3, 2026), designed for strong agentic, reasoning and conversational capabilities - optimized for complex multi-step agents, long-context analysis, and high-accuracy reasoning over code, math and science. It responds by first generating a reasoning trace and then concluding with a final response; reasoning is configurable through a chat-template flag.

It employs a hybrid Latent Mixture-of-Experts (LatentMoE) architecture - tokens are projected into a smaller latent dimension for expert routing and computation, improving accuracy per byte - with interleaved Mamba-2 and MoE layers plus select Attention layers, Multi-Token Prediction (MTP) layers using a shared-weight design across prediction heads (better training signal, native speculative decoding), 550B total / 55B active parameters, and a 262,144-token context extensible to 1M. It is pre-trained for ~20T tokens (crawled + synthetic code, math, science, general knowledge) with an NVFP4 quantization-aware recipe (majority of linear layers NVFP4; latent projections, MTP, QKV/attention and embeddings kept in BF16/MXFP8 for stability), then SFT on synthetic data and RL via asynchronous GRPO. Supports English, French, Spanish, Italian, German, Japanese, Hindi, Korean, Brazilian Portuguese, and Chinese; ready for commercial and non-commercial use.

Training Data Pre-trained for approximately 20T tokens using crawled and synthetic code, math, science, and general knowledge data. NVFP4 quantization-aware pre-training recipe. SFT on synthetic code, math, science, tool calling, instruction following, structured outputs. RL via asynchronous GRPO across math, code, science, instruction following, multi-step tool use, multi-turn conversations, and structured output environments.

Benchmark Scores

Benchmark Score Date Standing
Vals.ai Financial Agent 1.1 (without web search)
general_agent
88.00%
04.06.2026
#3/7
better than 67%
Terminal-Bench 2.1 (Terminus-2)
coding_agent
52.21%
04.06.2026
#29/44
better than 35%
CritPt (no tools)
stem_reasoning
7.89%
04.06.2026
#19/21
better than 10%
Vals.ai Financial Agent 1.1 (with web search)
general_agent
27.12%
04.06.2026
#6/7
better than 17%
GDPVal
general_agent
24.10%
04.06.2026
#19/20
better than 5%
IOI 2025
stem_reasoning
89.56%
04.06.2026
#3/5
better than 50%
MMLU-Pro
knowledge
86.45%
04.06.2026
#10/48
better than 81%
SWE-bench Verified
coding_agent
80.62%
04.06.2026
#37/60
better than 39%
SWE-bench Multilingual
coding_agent
74.20%
04.06.2026
#30/48
better than 38%
IMOAnswerBench
stem_reasoning
93.75%
04.06.2026
#7/30
better than 79%

Showing 10 of 50 results

What is a good Vals.ai Financial Agent 1.1 (without web search) score? What is a good Terminal-Bench 2.1 (Terminus-2) score? What is a good CritPt (no tools) score?

Hardware Requirements

Estimated VRAM per quantization at 4,096 tokens context — MoE: all experts must fit in VRAM, only 55.0B are active per token

Quantization Weights KV Cache Total VRAM On your hardware
FP16 — 16-bit 1100.0 GB 8.5 GB 1163.5 GB Which GPUs can run it? Calculate on your GPU
INT8 — 8-bit 550.0 GB 4.3 GB 581.8 GB Which GPUs can run it? Calculate on your GPU
INT4 — 4-bit 275.0 GB 2.1 GB 290.9 GB Which GPUs can run it? Calculate on your GPU
FP4 — 4-bit 275.0 GB 2.1 GB 290.9 GB Which GPUs can run it? Calculate on your GPU

Only datacenter GPUs — FP16 needs 1163.5 GB. For consumer hardware, choose the INT4 build with a multi-GPU setup.

Architecture

Decoder Block input Embedding vocab 131K · d 8192 Linear / Recurrent Hybrid 64:2 · dₕ 128 MoE FFN 512 experts · top-22 · +1 shared · dᴻ 5120 MTP Head ×1 speculative layer Final RMSNorm LM Head vocab 131K output
Attention
Hybrid Attention (64:2)
MoE
512 experts · top-22 per token
Hidden size
8192
Context
262K tokens
RoPE θ
10K
Parameters
550000M
Active params
55000M

Source: Hugging Face config.json · NemotronHForCausalLM · model repo

Type: Mamba2-Transformer Hybrid Latent Mixture of Experts (LatentMoE) with Multi-Token Prediction (MTP)
Attention: Hybrid (Mamba-2 SSM + select Attention layers)
Decoder: Mamba2-Transformer Hybrid
MoE: yes (? experts)
Routing: Latent routing - tokens projected into smaller latent dimension for expert routing
Mtp Design

Shared-weight design across prediction heads for speculative decoding

Training Recipe

NVFP4 quantization-aware pre-training

Training Pipeline

  1. 1
    pretraining

    Pre-Training

    Pre-trained for approximately 20T tokens using crawled and synthetic code, math, science, and general knowledge data. Training leveraged an NVFP4 recipe for efficiency. Software: Megatron-LM.

  2. 2
    sft

    Supervised Fine-Tuning (SFT)

    Fine-tuned on synthetic code, math, science, tool calling, instruction following, structured outputs, and general knowledge data. Incorporated data for long-range retrieval and multi-document aggregation. Data Designer used for corpus preparation.

  3. 3
    rl

    Reinforcement Learning (RL)

    Multi-environment RL using asynchronous GRPO (Group Relative Policy Optimization) across math, code, science, instruction following, multi-step tool use, multi-turn conversations, and structured output environments. Asynchronous RL architecture decoupling training from inference. Conversational quality refined through RLHF. Software: NeMo RL. Environments via NeMo Gym.

Training & Evaluation Datasets

NameRoleSizeModalitiesCollection
Nemotron Pre-Training Datasets pretraining — text —
Nemotron Post-Training v3 finetune — text —

Linked Resources

tech_report

NVIDIA Nemotron 3 Ultra Technical Report

https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf

 collection

NVIDIA Nemotron 3 model family on Hugging Face

https://huggingface.co/collections/nvidia/nvidia-nemotron-v3

 github

NeMo Evaluator SDK

https://github.com/NVIDIA-NeMo/Evaluator

 github

NeMo Gym (RL environments)

https://github.com/NVIDIA-NeMo/Gym

 github

NeMo Skills

https://github.com/NVIDIA-NeMo/Skills

 github

Harbor (evaluation framework)

https://github.com/harbor-framework/harbor

Comments (0)

No comments yet

Be the first to share your thoughts!

What now?

You have read the full card for NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 — here is where to go next.

Compare with another model Test on your own GPU Related models Data as JSON