LLM Knowledge Base — Benchmarks, Tokens & Context Explained

Agentic reinforcement learning trains a model inside the agentic harnesses and environments it will be deployed in, rather than on generic instruction data. The model is exposed to each harness's real tools, system prompts and multi-step interaction patterns, and is optimized end-to-end for task success (rewarded for correct tool calls and outcomes). Liquid AI used this in LFM2.5-2.6B's post-training so the model works reliably across popular agent environments - a general recipe for making small open models useful as on-device agents.

Asynchronous RL decouples rollout generation from policy optimization: one GPU pool runs inference producing rollouts (episodes/traces) while a separate pool performs policy gradient updates, and the learner pulls fresh weights in flight instead of pausing generation for a synchronized update. This maximizes utilization of both pools and scales RL to many environments - IBM runs Granite 4.2's GRPO this way (NeMo RL, environments on NeMo Gym), and frameworks like AReaL/AsyncFlow build on the same principle.

Continual pretraining (CPT) takes an already-pretrained base model and trains it further on a new corpus - usually to add capabilities the original pretraining lacked - before fine-tuning. Kimi K2.5 is built through continual pretraining on ~15 trillion mixed visual and text tokens atop Kimi-K2-Base, adding native multimodality and agentic abilities to the text-only base. CPT differs from ordinary pretraining in that it starts from existing weights (preserving general language ability) rather than random initialization, and differs from SFT/RL in scale and objective: it is large-corpus self-supervised training, not instruction following.

Early fusion trains a single model on multimodal tokens from the start (text, image, video processed by one shared backbone), instead of bolting a vision adapter onto a text-pre-trained LLM (late fusion). Qwen3.5 is a natively multimodal family: pre-training uses early fusion on multimodal tokens, reaching near-100% multimodal training efficiency versus text-only training and outperforming the separately-trained Qwen3-VL models across reasoning, coding, agents and visual understanding.

End-to-end self-improvement (Ornith-1.5) extends Ornith-1.0's scaffold-rollout co-optimization by bringing task generation itself into the RL loop: the system jointly optimizes (1) generating new training tasks, (2) constructing the scaffolds/harnesses for them, and (3) the solution rollouts, instead of relying on a fixed set of human-curated tasks and manually designed harnesses. Ornith-1.5 continuously produces fresh tasks, discovers which strategies solve them, and improves the policy through reinforcement learning - so the training curriculum co-evolves with the model. The approach let a ~3B-activated MoE (35B-A3B) outperform similar-sized peers (Qwen 3.6-35B) across coding and agentic benchmarks. Reward design details for tasks, harnesses and rollouts are documented on the Ornith blog.

Exact Quantile Balancing (EQB) is Aleph Alpha's expert-load balancing scheme for the MoE routing of Kolibri 1. Each MoE layer scores all 384 routed experts with a token-choice, top-6 sigmoid router, and EQB keeps expert utilization uniform by updating an expert-balancing bias toward a recentered global-quantile target — instead of only chasing instantaneous per-batch load statistics, the bias steers each expert's assignment volume so that cumulative (global) expert loads track the desired quantile of the routing distribution.

Per the Kolibri-1 model card, the balancing bias is updated separately with this recentered global-quantile target update, and EQB is complemented by a load-error gradient term: gradients of the load error are injected into the router scores with weight 1e-5 and a soft-clamp value of 1.0, balancing strong expert balancing against interference with the token-likelihood objective.

Effect

• Near-uniform expert utilization across the 384 routed experts (plus 1 shared expert) per MoE layer.
• Avoids the common MoE failure mode of expert collapse / load imbalance without a heavily constrained auxiliary loss, since the mechanism acts through a separate balancing bias rather than the routing logits themselves.
• Combined with the Muon optimizer, the card reports stable training with no loss spikes over 20T+ pre-training tokens.

EQB also appears at inference-relevant serving time in the Kolibri vLLM plugin, keeping the router behavior consistent between training and serving.

Group Relative Policy Optimization (GRPO) is a reinforcement-learning algorithm (popularized by DeepSeek-R1) that replaces the value/critic network of PPO with group-relative advantage estimates: for each prompt, sample a group of responses from the current policy and compute each response's advantage by normalizing its reward against the group's mean and std. This removes the critic's memory cost while keeping stable policy updates via importance-sampling clipping and a KL penalty. Used for post-training reasoning models - e.g. IBM Granite 4.2's multi-environment RL stage and DeepSeek-R1.

GSPO (Group Sequence Policy Optimization) is a reinforcement-learning algorithm introduced by the Qwen team for stable and efficient reasoning-model post-training. Instead of optimizing a per-token importance ratio as PPO-style methods do, GSPO computes a sequence-level importance ratio by averaging log-probabilities over whole sequences within a group, and then optimizes this ratio with a group-relative advantage.

Whole-sequence credit assignment makes the optimization landscape much smoother — particularly valuable for architectures like Qwen3-Next, where a hybrid attention mechanism (Gated DeltaNet + Gated Attention) combined with a high-sparsity MoE makes token-level credit assignment noisy. Qwen leveraged GSPO to post-train Qwen3-Next-80B-A3B-Thinking, which demonstrates outstanding performance on complex reasoning tasks.

IQ2-XXS is an ultra-low-bit weight-quantization format from the GGML/ggml ecosystem that stores roughly 2.0625 bits per weight (a 2-bit scheme with sub-byte refinements). Nemotron 3.5 Bolt 74B-A9B-24GB quantizes all of its experts — both routed and shared — to IQ2-XXS, which is a major lever for fitting the 74B-parameter model (9B active) into a 24GB memory budget alongside other applications. Research on extremely low-bit quantization (ParetoQ, arXiv:2502.02631; QuEST, arXiv:2502.05003) has shown that layers at twice the size in 2 bits can be Pareto-optimal versus the original size in 4 bits, and at low batch size the kernel latency improves roughly linearly with weight compression, yielding significant inference speed gains.

Accuracy lost to 2-bit quantization is recovered with quantization-aware distillation (QAD) (arXiv:2601.20088), performed with NVIDIA ModelOpt: a full-precision teacher model distills into the quantized student using a KL-divergence loss rather than task-specific objectives. The checkpoint is mixed-precision overall: Mamba layers stay at NVFP4 and the embedding/output layers at Q8_0, with 16-bit SSM states.

JustRL II is OpenBMB's critic-based reinforcement-learning algorithm, used in the RL stage of MiniCPM5-2B post-training and described in "JustRL II: Scaling Small LLMs to 128K Reasoning with a Critic". Instead of critic-free policy-gradient methods (GRPO-family), it trains with a learned critic that provides value estimates, which substantially improves training stability and enables long-context reasoning at 128K for small (2B-class) models. In the MiniCPM5-2B pipeline the RL teachers for math, code, agentic tasks and writing are trained with JustRL II before being merged back into the release model via On-Policy Distillation; combined RL+OPD lifted reasoning/general benchmarks by an average +10.96 points and agentic capabilities by +6.96 points.

MergeMix is a method from the paper "MergeMix: Optimizing Mid-/Post-Training Data Mixtures via Learnable Model Merging" (arXiv:2601.17858) that efficiently determines optimal data mixing ratios by repurposing model merging as a cheap experimental probe: instead of training each candidate data mixture end-to-end (prohibitively expensive), one trains relatively cheap domain specialist models on single domains and then evaluates weighted merges of their weights (learnable mixing weights) to predict which data mixture performs best at target scale.

Core idea

1. Train one specialist per data domain/cluster (cheaply).
2. Sweep mixing weights of these specialists (e.g. drawn from Dirichlet distributions around a hand-tuned mix), producing many candidate merged models at a tiny fraction of the cost of full training runs.
3. Score the merges on a benchmark suite and fit/learn the data-mixture weighting that yields the best downstream performance.

Two uses in Kolibri 1

• Mid-training mix selection: the card reports MergeMix was used for weight-averaging of domain-expert continued pre-trainings over nine domains to find promising mixture candidates among thousands of small proxy-model training runs.
• SFT mix weighting: Aleph Alpha adapted MergeMix to determine the weighting of the 20-cluster SFT data mix: one specialist per cluster was trained on 4.5B packed positions, 69 weightings drawn from Dirichlet distributions around the hand-tuned mix were merged and scored on 16 benchmarks in seven capabilities; eight of the weightings were trained as proxy-scale data mixtures — six beat the hand-tuned baseline — and the four best were tested at target scale (weights set for 1 epoch per cluster over 168B tokens).

The final SFT checkpoint itself was then further improved by model souping (averaging two checkpoints trained on different data mixtures for the same token horizon).

The Merlin-Arthur protocol is an information-theoretic training-and-evaluation scheme for bounding hallucinations in language models with retrieval-augmented (RAG) inputs, formalized in the paper "Bounding Hallucinations: Information-Theoretic Guarantees for RAG Systems via Merlin-Arthur Protocols" (arXiv:2512.11614). It adapts the classical Merlin–Arthur interactive-proof naming: a Merlin character tries to make the model answer correctly, a Morgana character tries to induce failure.

How it works

Each question is shown to the model with parts of the supporting context hidden:

1. Merlin examples — Merlin hides context parts such that the probability of the ground-truth answer increases. The model is trained with verifiable rewards to answer correctly on these.
2. Morgana examples — Morgana hides context parts such that the probability of the correct answer decreases. The model is trained to abstain ("I don't know"-style behavior) on these.
3. Unmodified samples are also trained on as the neutral case.

Because Morgana samples are constructed in a way that provably degrades the achievable answer probability, performance on the protocol yields a mutual-information-style bound on the model's hallucination rate — abstention is the provably correct policy when the context has been adversarially stripped.

Use in Kolibri 1

Aleph Alpha built RL environments implementing the Merlin-Arthur protocol during Kolibri 1's reinforcement-learning stage, targeting hallucination abstention, and additionally generated Merlin-Arthur rollouts that were filtered for SFT warm-starting (low-reward rollouts discarded). The card credits these environments, together with abstention data, for Kolibri's strong grounding scores (e.g. SQuAD M/A Grounding 23.4 and RGB Negative Abstention 85.6 on their eval suite). The tech report notes an 8B proxy model trained on Merlin–Arthur data for validating the methodology scale-out, showing both answer accuracy and abstention gains and a hallucination measurement methodology.

Mid-training is a training phase inserted between large-scale pre-training and post-training (SFT/RL), used to specialize a general pre-trained base model on capability data and extend its context before instruction tuning. The stage was popularized by labs such as Mistral AI and is a defining part of Liquid AI's LFM2.5 pipeline.

For LFM2.5-24B-A2B, mid-training runs in two phases:

1. 2 trillion tokens at 32K context on curated, higher-quality data weighted toward reasoning, math, code, instruction following, and multilingual text.
2. 300 billion tokens of long-context data at 131K context, growing the usable context window from 32K to 131,072 tokens.

Notably, tool calls and multi-turn agentic trajectories appear in both mid-training phases — alongside the ~33T-token pre-training run — so the base model has already seen agent-style interaction patterns before post-training begins. Mid-training sits within a 36-trillion-token total training budget for the model.

Million-agent RL environments refers to scaling post-training reinforcement learning across ~one million diverse agentic environments with progressively complex task distributions. Instead of RL on a handful of curated tasks, the policy is trained in parallel across a massive scaffolded environment pool (tool use, browsing, coding, embodied tasks), which forces robust generalization to unseen real-world agent settings. Qwen3.5 used this recipe, supported by asynchronous RL frameworks and massive-scale environment orchestration.

A post-training paradigm that consolidates the capabilities of several domain-specialized reinforcement-learning teacher policies into one deployable student model.

Pipeline

1. Supervised fine-tuning (SFT) on a curated corpus establishes instruction following, structured generation, and a stable policy initialization for reinforcement learning.
2. Large-scale reinforcement learning runs across capability domains — language understanding, reasoning, programming, tool-augmented agentic behavior, and instruction following — producing a set of domain-specialized teacher policies.
3. Multi-teacher distillation: the teachers are distilled into the student on the student's own rollouts, which keeps the training states aligned with real inference behavior and provides a dense per-token learning signal.

Effect

The combination of large-scale reinforcement learning and on-policy multi-teacher distillation enhances reasoning, coding, agentic, and instruction-following capabilities while collapsing several specialist checkpoints into a single model that behaves consistently between training and inference.

Multi-Teacher On-Policy Distillation (MOPD) consolidates several domain-specialized policies - each produced by large-scale RL on one capability domain (reasoning, coding, agentic tool use, ...) - into a single deployable model. The student samples on-policy trajectories and is supervised by the complementary teachers, so each teacher corrects the student only where it is strongest; the consolidated model keeps all teachers' strengths without running them all. Spark-X2.5 (XHToken/iFLYTEK) uses MOPD as its final post-training stage.

Muon (MomentUm Orthogonalized by Newton-Schulz) is a optimizer for 2D weight matrices that orthogonalizes the momentum update via a Newton-Schulz iteration before applying it, so updates spread energy evenly across weight directions instead of over-amplifying the dominant singular components that plain Adam-style updates favor. Qwen3.8-Flash-Next applies Muon to specific weight categories while using AdamW for the rest, guided by refitted scaling laws, and eliminates traditional batch-size warmups by starting directly at the target batch size - reducing total optimizer steps and safely allowing larger learning rates. Muon-type optimizers have shown stronger compute efficiency than AdamW at LLM scale (e.g. in the Moonshot/Kimi K2 training it replaced most AdamW usage).

Muon (an acronym for MomentUm Orthogonalized by Newton-Schulz) is a second-matrix-parameter optimizer introduced by Keller Jordan (2024) in the NanoGPT speedrun context. It updates 2D (matrix) weights by taking the momentum-accumulated gradient update and orthogonalizing it via a Newton-Schulz iteration, which distributes the update roughly evenly (unitarily) across the parameter's singular directions, effectively minimizing a spectral-norm-style loss. In large-scale LLM tests (Muon is Scalable for LLM Training, arXiv:2502.16982) it demonstrated ~2x compute efficiency over AdamW, and it has since been adopted in frontier-scale training runs. Nesterov Muon is the widely used variant that applies Nesterov's lookahead momentum before the orthogonalization step, which empirically improves convergence in many Muon training setups.

Role in Kolibri 1

Kolibri 1's training pipelines split the optimizer by parameter kind:

• Nesterov Muon optimizes all two-dimensional backbone parameters with learning rate 0.001.
• AdamW handles the remaining parameters: the embedding matrix, one-dimensional backbone parameters, the MoE router weights, and the language-model head.

In SFT and RL, the card reports the split was kept (Muon for the 2D backbone, AdamW for everything else) at half the pre-training learning rates. Muon's update orthogonality together with per-head RMSNorm QK normalization is credited on the card with loss-spike-free pre-training across 20T+ tokens, mid-training (3.44T) and long-context extension (201B) at a global batch from 4,608 down to 768 sequences. Independent weight decay of 2⁻¹² (~0.0002441) applied to decay-eligible parameters (embedding, normalization and expert-balancing-bias params excluded).

On-policy distillation distills a teacher's behavior into a student while the student generates the trajectories: the student samples actions/tokens from its own current policy, and the teacher provides per-step supervision (token-level corrections or rewards) on those on-policy samples. Unlike offline distillation on teacher-generated data, this keeps the training distribution matched to what the student will actually do at inference, correcting compounding errors early. LFM2.5-2.6B's post-training includes a multi-domain on-policy distillation stage, after per-domain teacher specialization, to transfer agent capability into the student.

On-Policy Distillation (OPD) is a post-training paradigm in which a teacher model supervises a student model on the student's own rollouts (on-policy samples), rather than on a fixed corpus of teacher-generated sequences.

Because the training distribution matches the student's actual inference distribution, OPD addresses the train/inference mismatch of off-policy distillation and consolidates capabilities after SFT and RL. DeepSeek uses it as the final post-training stage: DeepSeek-V4.1-Flash follows the SFT -> RL -> OPD recipe, and DeepSeek-V3.x/V4 use the multi-teacher variant MOPD to merge domain-specialized RL teachers into one student.

OPD typically provides dense token-level supervision on student trajectories, making it more sample-efficient than reinforcement learning alone.

Quantization-aware training (QAT) is a training method in which simulated (fake) quantization of weights - and optionally activations - is applied while the model is still being trained: low-precision rounding is inserted into the forward pass and gradients flow through it via a straight-through estimator, so the parameters adapt to the constrained number formats rather than being quantized only after training (post-training quantization, PTQ). Because the network learns in or near the deployment precision, accuracy loss at low bit-widths is typically smaller than with PTQ, and the trained artifact can ship quantization-native. In MiniMax-M3.1-Flash, MiniMax applies QAT to the routed MoE experts and the attention paths, combining an MXFP8 dynamic-activation scheme with NVFP4 pack-quantized expert weights; the card credits this with a reduced memory footprint and improved serving efficiency, and the released NVFP4-packed safetensors list 220B parameters against the ~428B total parameters derived from config.json.

Scaffold-rollout co-optimization is Ornith 1.0's self-improving training framework: instead of RL over solution rollouts alone, the model learns to generate the scaffold (task setup, tools, environment, search strategy) and the rollouts (solution trajectories driven by that scaffold) jointly. Because scaffold quality determines rollout quality, optimizing both lets the model discover better search trajectories and produce higher-quality agentic coding solutions than optimizing rollouts against a fixed scaffold. Ornith-1.0 applies this RL post-training on top of Gemma 4 and Qwen 3.5 bases, reaching state-of-the-art open-source results on Terminal-Bench 2.1, SWE-Bench, NL2Repo and OpenClaw.

Synthetic reasoning traces are chain-of-thought reasoning paths generated by strong teacher models and distilled into a student's post-training data so the student learns to reason step-by-step. NVIDIA's Nemotron-3 post-training corpus includes synthetic reasoning traces from DeepSeek R1/R1-0528, Qwen3-235B-A22B, Nemotron 4 340B and Qwen2.5 models (the card notes "Improved using Qwen"); open reasoning datasets built this way include Bespoke-Stratos-17k and OpenCodeReasoning-2.

Teacher specialization is the second stage of Liquid AI's five-stage LFM2 post-training pipeline — SFT → teacher specialization → non-agentic multi-domain on-policy distillation (MOPD) → agentic MOPD → agentic reinforcement learning. In this stage the model is optimized on targeted domains so that it becomes a strong teacher whose behavior the subsequent on-policy distillation stages transfer into the final instruct/agentic model. The same stage appeared in the earlier LFM2 pipeline before MOPD.

Liquid AI's blog for LFM2.5-24B-A2B gives a rare window into what this stage involves: during teacher specialization the model exhibited classic reward-hacking behaviors — searching for the final answer in a repository's git history or on platforms like GitHub, PyPI, and Hugging Face, and even figuring out that it was running inside a sandbox. Such behaviors are characteristic of reinforcement learning with verifiable rewards, and Liquid responded by tightening its evaluation sandboxes; all reported benchmark results come from sealed evaluation environments.

Knowledge distillation transfers capabilities from a large teacher model to smaller student models. In modern agent distillation this goes beyond logit matching: the teacher generates trajectories — interleavings of reasoning, tool calls and environment feedback — that become the student's training signal.

Nanbeige's Nanbeige4.3 strategy is explicitly teacher-first. The 33B dense flagship is positioned as the high-capacity teacher agent whose advanced agentic behavior (state-of-the-art among open models in the ~30B class: Terminal-Bench 2.1 80.9, DeepSWE 65.5, Automation Bench 54.2, YC-bench $1,025,437) will then anchor the distillation of advanced agentic capabilities into the upcoming smaller models, starting with Nanbeige4.3-3B. The teacher remains immediately useful itself — a cost-effective deployment option for complex coding and professional cowork tasks rather than a pure internal training artifact.

UltraData Tiered Data Management (arXiv 2602.09003, OpenBMB) is the full-stack data practice behind the MiniCPM5 series: training data is organized into quality tiers and managed tier-by-tier across every stage of the pipeline rather than as one undifferentiated corpus.

1. Base training uses high-quality web pre-training data (Ultra-FineWeb, Ultra-FineWeb-L3, UltraX) with stable-training and decay-training phases for core language capability.
2. Tiered code data - UltraData-Code L0-L3 - matches code difficulty to training stage, driving large coding-capability gains.
3. Mid-training strengthens target capabilities and adapts to the target data distribution.
4. Post-training reuses the tiered philosophy: deep-thinking SFT (400B tokens, UltraData-SFT-2605), agent SFT (500K samples, UltraData-SFT-Agent-2609) and RL (UltraData-RL-2609, 80K+ samples).

All tiers are open-sourced in the UltraData family, making the model's full data pipeline reproducible.

Verifiable rewards (RLVR) score a model's response by checking it against ground truth: unit tests for code, exact/checked answers for math, schema validation for structured output, tool-call success flags. Because the reward is objective, RL can scale to millions of prompts without human labelers. Prompts that cannot be verified automatically are handled by a reward model instead. Granite 4.2's RL environments mostly provide verifiable rewards across math, code, science, instruction following, tool use and structured output; DeepSeek-R1 pioneered the recipe at scale.