NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB

Architecture

Training Pipeline

1. 1
   cpt

   Stage 1: Compression (Star Elastic)

   The Nemotron 3.5 Super VL 120B-A12B parent checkpoint is compressed to Bolt 74B-A9B using Star Elastic, NVIDIA's push-button, router-guided architecture search and end-to-end compression algorithm. A budget-conditioned router heterogeneously selects routed-expert counts and MoE intermediate widths across layers, jointly optimizing architecture and weights through knowledge distillation with a model-size penalty, targeting a deployment footprint below 24GB at batch size 1 and 128K context. Architecture search runs at 8K context with the router converging during the first 8B tokens; recovery continues at 8K to a total of 72B tokens, followed by 48B tokens at 128K context, for 120B tokens overall.

2. 2
   sft

   Stage 2: Vision Adaptor Training

   The trained vision tower from the parent Super 3.5 VL model is re-attached to the compressed LLM, followed by lightweight fine-tuning of only the vision-to-language projection layers (the vision adapter), using image-text datasets previously used to train the parent model's vision adapter. Training used an image-only blend of approximately 6.6M samples per epoch, comprising a general English mix of captioning, visual question answering, counting, and document datasets (~2M samples), CC3M images recaptioned by Qwen3-30B (~1.7M samples), and Chinese document OCR data (~2.5M samples), plus PDF/table document data and NVCLIP OCR datasets. No video or text-only data was used during vision adapter training.

3. 3
   sft

   Stage 3: Supervised Fine-Tuning

   After vision-adaptor training, the model is trained with vision-heavy SFT to further recover vision capabilities; before multi-turn agentic training, checkpoint merging was applied for more well-rounded text and vision capabilities, and a final checkpoint merge after agentic-environment and RLHF training yielded the final BF16 checkpoint. Vision blends target ScreenSpot Pro (~10.1B tokens) and WebArena (~9.5B tokens), supplemented with MolmoWeb/WebVoyager browsing data, CharXiv chart data, HopChain multi-page document data, and BabyVision-targeted reasoning data; text uses a downsampled blend of Kimi-K3-generated synthetic datasets targeting AIME, GPQA Diamond, Terminal Bench 2.1, SciCode, AA-LCR, APEX, and LMArena, plus a small proportion of GLM 5.2-generated MCP-Atlas trajectories. During this stage the LLM, vision-adaptor, and vision encoder were all trained, and the vision-heavy SFT checkpoint was merged with the vision-adaptor-only checkpoint via SLERP interpolation.

4. 4
   rlhf

   Stage 4a: Reinforcement Learning from Human Feedback (RLHF)

   As part of Stage 4, the model was further trained with RLHF to improve alignment, using the Nemotron 3 Ultra GenRM model as reward model, which was itself trained with a reasoning penalty for token efficiency. This RLHF pass ran alongside several stages of E2E RL training to improve agentic capabilities and minimize task completion times for tool-heavy multi-turn workloads.

5. 5
   rl

   Stage 4b: E2E GRPO Reinforcement Learning (MCP and SWE)

   The model was trained with full E2E GRPO on multiple agentic environments targeting MCP and SWE tasks: the MCP environment uses a simulated MCP-server with multi-turn tool interactions (32 prompts x 16 generations = 512 batch size) and improved MCP-Atlas from ~48 to ~56 on the 500 public task subset. SWE-RL E2E training used 64 prompts with 16 generations per prompt (1,024 trajectories per optimizer step) at a maximum total sequence length of 196,608 tokens, with Adam at a learning rate of 3e-6 and optimizations for generated-token and tool-call efficiency. The Terminal environment was not used for RL training but generated on-policy rollouts to prevent regressions during quantization; the RL training also generalized to PinchBench scores up to 90.4 across a 10-repeat average.

6. 6
   other

   Stage 5: Quantization (QAD)

   The experts of the model are quantized to 2 bits to fit in RTX Spark's memory, using quantization-aware distillation (QAD) to recover accuracy, performed with NVIDIA ModelOpt. All experts (routed and shared) are quantized to IQ2-XXS at 2.0625 bits per element, a lever for the for-size checkpoint since lower-bit FC layers can be Pareto-optimal and low-bit weights improve kernel latency at low batch sizes. Further mixed precision keeps Mamba layers at NVFP4 and embedding/output layers at Q8_0 to meet the 24GB memory constraint without impacting capabilities.

Training & Evaluation Datasets

Linked Resources