LLM Knowledge Base — Benchmarks, Tokens & Context Explained

Agent Swarm is Kimi K2.5's self-directed, coordinated execution scheme: instead of scaling a single agent loop, the model decomposes a complex task into parallel sub-tasks and dynamically instantiates domain-specific agents to execute them in a swarm-like fashion. Each specialized agent handles its sub-task (e.g. visual data processing, coding, tool use), and execution is coordinated back into one coherent result. This shifts the scaling axis from single-agent chain length to parallel agent populations, improving throughput on large multi-step tasks and grounding each sub-task in an agent tuned for that domain.

Agentic reinforcement learning trains a model inside the agentic harnesses and environments it will be deployed in, rather than on generic instruction data. The model is exposed to each harness's real tools, system prompts and multi-step interaction patterns, and is optimized end-to-end for task success (rewarded for correct tool calls and outcomes). Liquid AI used this in LFM2.5-2.6B's post-training so the model works reliably across popular agent environments - a general recipe for making small open models useful as on-device agents.

Asynchronous RL decouples rollout generation from policy optimization: one GPU pool runs inference producing rollouts (episodes/traces) while a separate pool performs policy gradient updates, and the learner pulls fresh weights in flight instead of pausing generation for a synchronized update. This maximizes utilization of both pools and scales RL to many environments - IBM runs Granite 4.2's GRPO this way (NeMo RL, environments on NeMo Gym), and frameworks like AReaL/AsyncFlow build on the same principle.

Attention Residuals (AttnRes) is one of the two architectural components Moonshot AI built Kimi K3 on (with Kimi Delta Attention). Standard transformers add each layer's block output through a residual stream that bypasses attention; AttnRes instead feeds information from the attention pathway itself back through the residual stream - the residual connection carries attention's contribution rather than only the MLP/block output. Combined with delta-rule linear attention (KDA) and a Stable LatentMoE backbone (16 of 896 experts active), this is part of what yields Kimi K3's ~2.5x improvement in overall scaling efficiency over Kimi K2.

C-RADIO is NVIDIA's "Commercial" line of agglomerative vision foundation models, derived from the AM-RADIO approach ("Reduce All Domains Into One"). Agglomerative models distill multiple state-of-the-art vision teachers — for C-RADIOv4, teachers such as SigLIP2, DINOv3, and SAM3 — into a single student backbone, unifying their capabilities. The encoder returns both dense spatial features (cleaner object boundaries in v4 than v3) and summarization tokens (analogous to a ViT class token) that represent the general content of an image, making the features suitable for both classification-style and dense prediction-style downstream use.

Nemotron 3.5 Bolt 74B-A9B uses the C-RADIO v4-H vision encoder to ingest image and video frames, connected to the LLM backbone through an MLP projector.

Causal Encoder-Decoder (CED) is the backbone architecture of the DeepSeek-V4.1-Flash platform, carried over unchanged into the 1017 checkpoint: a 40-layer Transformer split into a 20-layer causal encoder and a 20-layer decoder. During prefill (about 8B of the 763B MoE parameters active per token) the causal encoder processes text and vision inputs; the decoder's global KV cache is then projected once from the final encoder hidden states rather than being derived from each decoder layer's own hidden states, and that single compact projection is reused by all decoder layers. Because the decoder keeps its KV in this small, regular encoder-derived form, the platform's other mechanisms — Compressed Sparse Attention 2 (CSA2) cross-layer reuse over the decoder KV, FP4 main KV caching, and deployment-side SWA Bounded Replay for sliding-window layers — can compress it aggressively: the V4.1-Flash tech report measures the global KV cache at 890 bytes per token, roughly 1/4 of DeepSeek-V4-Flash and about 437× smaller than DeepSeek-V1.

Causal Encoder-Decoder (CED) is a Transformer architecture introduced with DeepSeek-V4.1-Flash that splits the network into a causal encoder followed by a decoder. Instead of deriving the decoder's global KV cache from each decoder layer's own hidden states, CED projects it from the final encoder hidden states.

This makes prefill dramatically cheaper: DeepSeek-V4.1-Flash (552B backbone parameters) activates only 8B parameters per token during prefill and 16B during decode, which is particularly cost-efficient for input-heavy agentic workloads where long prompts are processed once and short answers are generated.

The 40-layer model uses a 20-layer causal encoder plus a 20-layer decoder. Combined with CSA2 attention and FP4 KV caching, CED is one of the four techniques that cut the model's global KV cache to 890 bytes per token (~1/4 of DeepSeek-V4-Flash).

Also called autoregressive or decoder-only language modeling: the model predicts the next token in a sequence using only the tokens that precede it. Generation runs strictly left-to-right and never conditions on future tokens.

Core principle

Given a sequence of tokens x1, x2, ..., x_{t-1}, the model estimates the conditional probability of the next token:

P(x_t | x1, x2, ..., x_{t-1})

The full sequence probability is the product of these conditional probabilities (chain rule of probability):

P(x1, ..., x_T) = product over t of P(x_t | x1, ..., x_{t-1})

Causal masking

This unidirectionality is enforced architecturally through a triangular attention mask. In standard self-attention, every token can attend to every other token. Causal masking restricts this so that position t can only attend to positions 1 through t. The mask is an upper-triangular matrix filled with -infinity values, applied to attention scores before softmax.

Key distinctions

• Causal (decoder-only) LM: lower-triangular mask, sees only the past, generates left-to-right.
• Masked LM: full bidirectional attention, sees the entire input at once, used for understanding tasks.
• Prefix LM: hybrid — bidirectional attention on a prefix portion, autoregressive generation on the remainder.

References

• Vaswani et al. (2017), "Attention Is All You Need", arXiv:1706.03762
• Hugging Face Transformers Documentation: Causal language modeling
• AI Wiki: https://aiwiki.ai/wiki/causal_language_model

Compressed Sparse Attention (CSA), introduced with the DeepSeek-V4 series, is one half of the family's hybrid long-context attention (paired with Heavily Compressed Attention, HCA). CSA sparsifies which positions a query attends to while compressing the retained key/value representations, so both the search over the context and the cache that backs it shrink. DeepSeek reports that at 1M-token context the CSA/HCA hybrid lets DeepSeek-V4-Pro run with only 27% of single-token inference FLOPs and 10% of the KV cache versus DeepSeek-V3.2 - the enabling factor behind the series' native 1M context.

Compressed Sparse Attention 2 (CSA2) is the sparse-attention scheme of the DeepSeek-V4.1-Flash platform, extending the Top-K sparse attention (DSA) introduced with DeepSeek-V3.2. Every attention layer is statically assigned exactly one of three modes: Full generates its own main KV and indexer keys and computes fresh Top-K indices; Reindex reuses a preceding layer's shared main KV and indexer keys but re-scores the candidate pool to select its own Top-K indices; Reuse shares both the KV blocks and the Top-K indices of an earlier layer. A Hierarchical Sparse Indexer — Top-K of 512 over a bounded candidate pool, plus a 128-token sliding window — keeps indexer cost roughly independent of context length. Because main KV and indexer state are shared across layers, the shared KV is stored in FP4, and the first encoder layers retain local sliding-window attention (SWA), the global KV cache (kept in HBM) shrinks to 890 bytes per token, roughly 1/4 of DeepSeek-V4-Flash.

Compressed Sparse Attention 2 (CSA2) is the attention scheme of DeepSeek-V4.1-Flash. Each attention layer is assigned one of three static modes:

• Full - standard full attention; the first Full Mode layer also builds the candidate pool for later indexing layers
• Reindex - recomputes attention indices from shared state
• Reuse - reuses Top-K sparse-attention indices computed by another layer

Layers share main KV and indexer K across each other, and a Hierarchical Sparse Indexer restricts later indexing layers to the candidate pool constructed by the first Full Mode layer, bounding deeper indexer cost independently of context length.

Together with FP4 main KV caching, CSA2 reduces the global KV cache footprint to 890 bytes per token - roughly 1/4 of DeepSeek-V4-Flash - while supporting 1M-token contexts.

Configurable reasoning effort in the gpt-oss models (120b and 20b) lets the caller choose how much thinking the model does per query - low, medium, or high - via a simple system prompt (Reasoning: low / medium / high) or the reasoning_effort API parameter. Lower effort answers faster with less chain-of-thought; higher effort spends more tokens reasoning before answering. The full chain-of-thought is always exposed to the developer (for debugging and trust; OpenAI notes it is not intended to be shown to end users), and the model is trained to co-exist with tool calls inside that reasoning stream. This native effort control is one of gpt-oss's signature features and the reason latency-sensitive agentic deployments pick low/medium while hard math picks high.

Continual pretraining (CPT) takes an already-pretrained base model and trains it further on a new corpus - usually to add capabilities the original pretraining lacked - before fine-tuning. Kimi K2.5 is built through continual pretraining on ~15 trillion mixed visual and text tokens atop Kimi-K2-Base, adding native multimodality and agentic abilities to the text-only base. CPT differs from ordinary pretraining in that it starts from existing weights (preserving general language ability) rather than random initialization, and differs from SFT/RL in scale and objective: it is large-corpus self-supervised training, not instruction following.

Controllable Thinking Budget is Spark-X2.6-4B's inference-time control over how much explicit reasoning the model performs before answering. The model provides three configurable thinking levels - none, low, and high - allowing applications to dynamically balance reasoning quality, latency, and inference cost instead of always paying for deep reasoning. Thinking is enabled by default by the chat template, and can be turned off per request with the enable_thinking toggle passed via chat_template_kwargs (for example {"enable_thinking": false} against an OpenAI-compatible server); when serving with SGLang, the Qwen3 reasoning parser separates the thinking section from the final answer. Per-request thinking-effort switches are becoming a common pattern in modern LLMs, and Spark's three levels make on-device deployments tunable per task: quick lookups can run with thinking disabled, while hard tool-use, planning, or coding problems can run with deeper thinking.

Cross-layer KV cache sharing lets consecutive Transformer layers reuse the same keys/values instead of each layer computing and storing its own KV cache - the attention outputs of one layer's projection serve its neighbors. In Gemma 4 E-models, 18 of the layers share KV entries (config num_kv_shared_layers: 18), directly shrinking the KV cache that dominates long-context memory on edge devices. Combined with sliding-window attention, it keeps the E4B's 128K context practical on phones and laptops.

A decision model is a language model built to answer a fixed schema of typed questions about an input state — classification, scoring, routing, gating — rather than to generate free-form text. An emerging decision-model infrastructure category wraps weights, APIs and eval suites around fast, structured, typed answers.

Key properties, exemplified by Cloudflare's Clef:

• Input: a state (text, JSON, images, video) plus a schema of typed questions with enumerated options.
• Output: a probability for every allowed option of every question — computed in a single forward pass, with no token generation and no output parsing.
• Latency profile: because decoding is not autoregressive, cost is close to one prefill pass over the state. On the card's Decision Index run, Clef's median request latency is 209.3 ms vs 524.1 ms for the generative sibling Jev, and Cloudflare reports up to 13x speedups for the Clef-flash variant.
• Training: post-trained from general backbone LLMs (Clef from Qwen3.8-27B), typically with a specialized scoring head on top.

Examples in this family: TypeSafe's Jev, Cloudflare's Clef and Clef-flash, and AWS's Strands Decider 2B (all late 2026). The trade-off vs generative LLMs: decision models return schema-bounded, calibrated answers at high throughput, but they do not explain or generate — they decide.

References: Clef model card, Cloudflare blog, The Register, Marktechpost.

DeepSeek-V4-Pro-Max is the maximum reasoning effort mode of DeepSeek-V4-Pro, announced with the DeepSeek-V4 preview series. DeepSeek-V4 models expose configurable reasoning effort; at the Max setting the model spends the largest thinking budget, which significantly advances the knowledge capabilities of open-source models - top-tier performance in coding benchmarks and a significantly narrowed gap with leading closed-source models on reasoning and agentic tasks, establishing it (per DeepSeek) as the best open-source model available at release. The same mechanism gives DeepSeek-V4-Flash-Max comparable reasoning performance to the Pro version when given a larger thinking budget, though its smaller scale still trails on pure knowledge tasks and the most complex agentic workflows. Related: gpt-oss models expose a similar configurable-effort control (low/medium/high).

DeepStack is a vision-language fusion mechanism used in Qwen3-VL: instead of feeding only the final ViT layer into the language model, DeepStack injects multiple intermediate ViT feature maps into several early LLM layers through specialized mergers. Fusing multi-level visual features captures fine-grained details and sharpens image-text alignment. In Qwen3-VL-30B-A3B the vision encoder is 27 layers deep and the DeepStack visual indexes are layers 8, 16 and 24. The Qwen3-VL Technical Report ablates DeepStack with a 15B-A2B model, showing clear gains from multi-level feature fusion.

DFlash attention is the DeepSeek-V4 family's attention mechanism (used in DeepSeek-V4-Flash and the Flash-Vision-Exp multimodal variant). It combines latent-compressed queries and outputs (q/o LoRA ranks of 1024, outputs factored into 8 groups) with a learned sparse index: a separate set of 64 index heads (dim 128) scores every position and keeps only the top-512 entries per query token for the full attention computation, while 3 hash layers and a 128-token sliding window provide additional locality. Full attention runs with 64 heads of head dim 512 but only 1 KV head. The result is near-linear cost on very long sequences while preserving retrieval quality, which together with YaRN scaling supports DeepSeek-V4's 1M-token context.

DFlash (arXiv 2602.06036) is a block-diffusion model used as a lightweight speculative-decoding drafter: instead of predicting one token at a time, it proposes entire blocks of tokens (e.g. 16 tokens) in a single forward pass, which the main model then verifies in parallel - accepting correct tokens and correcting wrong ones. Because verification is parallel, output quality is identical to standard autoregressive decoding while generation is significantly faster. Muse-Glimmer-30B ships with a DFlash drafter (5 draft layers, 16-token blocks, sliding-window attention with 2048 window on all layers, 32 Q / 8 KV GQA heads, sequence length 131,072, hidden features drawn uniformly from target layers {1, 13, 25, 37, 49} of 52), delivering measured speedups of 3.1x on an NVIDIA RTX 5090, 1.8x on Apple M5 Max and 1.5x on M4 Max with quantized drafters available to cut memory overhead. Distinct from the DeepSeek-V4 "DFlash attention" sparse-attention mechanism despite the shared name.

DFlash2 is the speculative-decoding draft model KRAFTON ships alongside Raon-1 mini preview (repository KRAFTON/Raon-1-mini-preview-dflash2). It builds on DFlash — "Block Diffusion for Flash Speculative Decoding" (z-lab; arXiv:2602.06036), a drafting framework that replaces a conventional autoregressive drafter with a lightweight block-diffusion model: the drafter is conditioned on context features fused from several target-model layers and predicts an entire block of tokens in a single forward pass, while the target model verifies every proposed token losslessly.

In KRAFTON's benchmarks on a single NVIDIA B300 (vLLM, MXFP8 target, block size 8 → 7 draft tokens, probabilistic draft sampling, thinking enabled), DFlash2 delivers up to 2.18× higher output throughput than autoregressive decoding, with acceptance lengths of roughly 3.2–4.8 tokens per verification step across GSM8K, MATH-500, HumanEval, MBPP, and MT-Bench. KRAFTON serves it via a vLLM fork (--spec-method dflash --spec-tokens 7) and llama.cpp (--spec-type draft-dflash).

Double-gated convolution blocks are the non-attention half of Liquid AI's LFM2/LFM2.5 hybrid architecture. Instead of attention, a token block passes through depthwise causal convolutions whose inputs and outputs are controlled by two learned gates (an input gate and an output gate), giving a fixed-size local receptive field (conv window L=3 in LFM2.5) with constant memory and compute per token. LFM2.5-8B-A1B uses 18 such conv blocks interleaved with only 6 GQA attention layers (ratio 3:1), which keeps the long-context KV-cache and attention FLOPs small - the key to its on-device throughput.

DSpark is DeepSeek's speculative-decoding engine, integrated into DeepSeek-V4.1-Flash(-1017) as one of its efficient architectural extensions (published as the DSpark paper on confidence-scheduled speculative decoding). It pairs a small semi-autoregressive drafter with a confidence head and a hardware-aware scheduler: the parallel drafter proposes long token blocks in a single forward pass, the confidence schedule decides at each step how much of the block is worth verifying, and target-model verification discards any non-matching tokens — so the output remains byte-for-byte identical (lossless) to plain decoding. DSpark addresses the rapid acceptance decay that indiscriminate parallel drafters suffer, where the acceptance probability of long proposed blocks falls off quickly and verification compute is wasted. DeepSeek reports roughly 57-85% faster inference per user and up to about 6.6x higher throughput on V4-class models, and third-party tooling (llama.cpp guides, NVIDIA NeMo recipes) documents training and using DSpark drafters for DeepSeek V4.1 Flash.

DSpark is a component of the DeepSeek-V4 forward path (named in the DeepSeek-V4-Flash-Vision-Exp reference inference). Its config keys mark a parallel token stream: dspark_block_size: 5, dspark_markov_rank: 256, dspark_target_layer_ids: [40, 41, 42] (the final three of 43 layers), and a dedicated dspark_noise_token_id. Together with num_nextn_predict_layers: 3 (multi-token prediction) this forms DeepSeek-V4's speculative/parallel generation machinery: blocks of up to 5 tokens are processed through a Markov-rank-256 state on the last layers with a noise token controlling stochastic exploration. The public reference implementation documents its forward path; DeepSeek has not yet published a full paper on DSpark.

DSpark is a confidence-scheduled speculative decoding method in which a lightweight draft model proposes whole blocks of tokens in parallel while a frozen target model verifies them exactly. Mistral Large 4 (Le Chonk) is notable for shipping an embedded DSpark draft head inside the checkpoint: 3 dense MLA draft layers in BF16 that use a recurrent Markov head to propose a block of 3 tokens per step. Because the target model still verifies every draft token, DSpark accelerates decoding without changing the target model's outputs; Mistral measured up to 1.9x higher throughput than serving the target model alone, with gains depending on hardware and workload. It is enabled with --speculative-config '{"method": "dspark", "num_speculative_tokens": 3}' on vLLM or --speculative-algorithm DSPARK on SGLang, where num_speculative_tokens: 3 matches the draft-block depth.

Research note (added facts, not from the card): public sources describe DSpark (arXiv:2607.05147, titled "DSpark: Confidence-Scheduled Speculative Decoding") as extending the DFlash block-parallel drafter with two heads — a low-rank Markov head that restores intra-block token dependency (mitigating the acceptance decay of purely parallel drafters) and a confidence head predicting per-position acceptance probability; NVIDIA NeMo/Megatron-Bridge document training DSpark drafters against a frozen target, and the vLLM speculators project documents the same algorithm. The Mistral card itself names DSpark and the Markov head but not DFlash, the confidence head, or the paper.

Early fusion trains a single model on multimodal tokens from the start (text, image, video processed by one shared backbone), instead of bolting a vision adapter onto a text-pre-trained LLM (late fusion). Qwen3.5 is a natively multimodal family: pre-training uses early fusion on multimodal tokens, reaching near-100% multimodal training efficiency versus text-only training and outperforming the separately-trained Qwen3-VL models across reasoning, coding, agents and visual understanding.

Encoder-free multimodal architecture (Gemma 4 12B "Unified") removes the dedicated vision/audio encoders that most multimodal LLMs bolt onto a text backbone. Instead, raw image patches and audio waveforms are projected directly into the LLM's embedding space through lightweight linear layers, so every modality flows into a single decoder-only transformer. This reduces multimodal latency, simplifies the pipeline (no separate ViT/AST to download or run), and allows the entire model to be fine-tuned in one pass - at the cost of putting all cross-modal work on the LLM itself.

End-to-end self-improvement (Ornith-1.5) extends Ornith-1.0's scaffold-rollout co-optimization by bringing task generation itself into the RL loop: the system jointly optimizes (1) generating new training tasks, (2) constructing the scaffolds/harnesses for them, and (3) the solution rollouts, instead of relying on a fixed set of human-curated tasks and manually designed harnesses. Ornith-1.5 continuously produces fresh tasks, discovers which strategies solve them, and improves the policy through reinforcement learning - so the training curriculum co-evolves with the model. The approach let a ~3B-activated MoE (35B-A3B) outperform similar-sized peers (Qwen 3.6-35B) across coding and agentic benchmarks. Reward design details for tasks, harnesses and rollouts are documented on the Ornith blog.

Engram conditional memory is DeepSeek's second axis of sparsity beside MoE routing in DeepSeek-V4.1-Flash(-1017): rather than activating a mixture of experts, specific layers of the backbone attach a conditional-memory module that is read by n-gram lookup. At each position, the few preceding tokens are hashed into n-gram keys and the matching rows are gathered from large read-only Engram tables into the layer's computation — in serving discussions characterized as a token-hash row-gather (a lookup, not a GEMM), so the cost stays low and essentially independent of context length. The idea modernizes classic n-gram statistics into a conditional-memory component for LLMs (described in follow-up literature as a complementary axis of sparsity, e.g. conditional memory via scalable lookup); because natural-language n-grams follow a Zipfian distribution, a small hot fraction of table entries serves most accesses. The vLLM serving stack exposes Engram as a first-class feature (engram), rejecting the configuration for models without n-gram embeddings.

Exact Quantile Balancing (EQB) is Aleph Alpha's expert-load balancing scheme for the MoE routing of Kolibri 1. Each MoE layer scores all 384 routed experts with a token-choice, top-6 sigmoid router, and EQB keeps expert utilization uniform by updating an expert-balancing bias toward a recentered global-quantile target — instead of only chasing instantaneous per-batch load statistics, the bias steers each expert's assignment volume so that cumulative (global) expert loads track the desired quantile of the routing distribution.

Per the Kolibri-1 model card, the balancing bias is updated separately with this recentered global-quantile target update, and EQB is complemented by a load-error gradient term: gradients of the load error are injected into the router scores with weight 1e-5 and a soft-clamp value of 1.0, balancing strong expert balancing against interference with the token-likelihood objective.

Effect

• Near-uniform expert utilization across the 384 routed experts (plus 1 shared expert) per MoE layer.
• Avoids the common MoE failure mode of expert collapse / load imbalance without a heavily constrained auxiliary loss, since the mechanism acts through a separate balancing bias rather than the routing logits themselves.
• Combined with the Muon optimizer, the card reports stable training with no loss spikes over 20T+ pre-training tokens.

EQB also appears at inference-relevant serving time in the Kolibri vLLM plugin, keeping the router behavior consistent between training and serving.

FFN activation clipping clamps the feed-forward network's intermediate activations to a bounded magnitude - in Ripple-65B, ffn_clamp_threshold: 20.0 - so oversized hidden values cannot accumulate through a deep stack. Akin to the attention-logit soft-capping used in Gemma 2 but applied to the FFN path, activation clipping is used in recent frontier pretraining recipes (e.g. Step 3.5 Flash) to keep internal-norm trajectories stable and reduce loss spikes. This matters for Ripple-65B's thin-and-deep design (80 layers, d_model 5120, FFN intermediate size 40960), trained in plain BF16 with no custom kernels.

FP4 main KV caching stores the attention KV cache in 4-bit floating point format (E2M1: 1 sign bit, 2 exponent bits, 1 mantissa bit) with one E4M3 scale factor per 16 channels, a block-scaled scheme in the spirit of NVFP4.

In DeepSeek-V4.1-Flash, FP4 main KV caching works together with CSA2 attention and SWA Bounded Replay to shrink the global KV cache to 890 bytes per token - roughly 1/4 of DeepSeek-V4-Flash and 1/437 of DeepSeek-V1 - a decisive cost factor for long-context agentic workloads where KV cache memory dominates serving cost.

FP8 is an 8-bit floating-point format (commonly the E4M3 and E5M2 encodings) that halves per-element memory versus BF16 and benefits from native FP8 tensor-core acceleration on Hopper/Blackwell GPUs, making it a standard choice for frontier-scale checkpoints and serving. The Mistral Large 4 checkpoint ships FP8-quantized weights with a dynamic activation scheme; per the model card, sensitive components stay at higher precision — embeddings, the vision tower, the projector, lm_head, and the KV/index projections remain in BF16/F32. Mistral also publishes a separate NVFP4 checkpoint (4-bit floating-point format targeting NVIDIA Blackwell GPUs) with a smaller memory footprint and faster inference at the same performance.

Gated Attention is the full-attention branch of Qwen3-Next's hybrid attention layout. Unlike standard Transformer attention, it applies a learnable output gate to each attention head, which stabilizes training and improves long-context quality while retaining exact token-to-token attention.

In Qwen3-Next-80B-A3B-Thinking, Gated Attention layers are interleaved with Gated DeltaNet layers in a 12 x (3 x (Gated DeltaNet -> MoE) -> 1 x (Gated Attention -> MoE)) pattern. Each Gated Attention block uses 16 query heads and 2 KV heads with a head dimension of 256 and a rotary position embedding dimension of 64.

The gated design keeps the precise-retrieval benefits of full attention while adding the training stability that hybrid linear-attention models need for robust pre-training and post-training (together with zero-centered and weight-decayed layernorm).

Gated Attention is an attention-block design that adds a learned multiplicative gate — and in some variants an additional normalization — on the output of standard dot-product attention, letting the network suppress or amplify attention results before they re-enter the residual stream. In modern implementations (e.g. Qwen3-Next), the scaled dot-product attention output is multiplied by a sigmoid gate computed from the layer input before the output projection, and the block also uses a zero-centered QK-Norm variant to stabilize query/key logits.

Muse Spark Open's language model enables Gated Attention with an output gate and an output norm on the attention block, plus QK-norm on queries and keys. Together with grouped-query attention (128 Q heads / 16 KV heads, head dimension 64) across its 62 layers, this stabilizes training and inference of the 816B-total / 42B-active MoE transformer.

Gated DeepSeek Sparse Attention (Gated DSA) is the attention module of Tencent's Hy4 preview (Hy series), adapted from DeepSeek's Sparse Attention (DSA, arXiv 2512.02556) with a gating mechanism, itself inspired by attention work in DeepSeek and GLM models. A lightweight indexer produces a sparse candidate set per query and attention runs only over those positions, cutting compute at very long context; Hy4 pairs it with IndexCache, which reuses the sparse index across layers. Hy4-preview configuration: 64 attention heads, query compression dimension 2048, key-value compression dimension 512, indexer with 32 heads / 128 head dimension and top-k 2048. Related but distinct: DeepSeek-V4's Compressed Sparse Attention (CSA) and DeepSeek-V4.1-Flash's CSA2 are the DeepSeek family's own sparsified attention schemes - DSA is the earlier scheme Hy4 gates and borrows.

A linear attention architecture that combines data-dependent gating with the delta update rule to achieve efficient, high-quality sequence modeling. Introduced by NVIDIA Research (NVlabs) as an improvement over Mamba2 and DeltaNet.

Background: linear attention

Standard Transformer attention scales quadratically with sequence length L. Linear attention replaces softmax with a kernelized dot product and reframes computation as a linear RNN with a matrix-valued state S_t, reducing inference memory to O(1) per step regardless of sequence length.

Delta rule

DeltaNet uses the delta update rule (Widrow et al., 1960): instead of accumulating all key-value pairs into the state (Hebbian learning), it selectively replaces the value associated with the current key. This is more surgical — it only modifies one key-value pair at a time while leaving others intact. The update is:

S_t = S_{t-1}(I − beta_t * k_t * k_t^T) + beta_t * v_t * k_t^T

This gives superior associative recall but lacks a mechanism to rapidly clear outdated information during context switches.

Gated delta rule

Unifying gating and the delta rule gives:

S_t = S_{t-1} * (alpha_t * (I − beta_t * k_t * k_t^T)) + beta_t * v_t * k_t^T

• alpha_t (gate, 0 to 1): controls state decay. Setting alpha_t → 0 flushes the entire state (rapid forgetting); alpha_t → 1 reduces to the pure delta rule (selective update).
• beta_t (writing strength, 0 to 1): controls how much of the new value is written.

This combination enables both bulk forgetting (via gating) and targeted updates (via delta rule), giving flexible memory control.

Hardware-efficient training

The recurrence is sequential, but a chunkwise parallel algorithm (using the WY representation and matmuls) enables GPU-efficient training with tensor cores. Within each chunk, the sequential dependencies are captured by a small C x C matrix inverse (C = chunk size).

Hybrid architectures

The layer is often combined with sliding-window attention (SWA) or Mamba2 layers in hybrid models. For example, a layout of 16 blocks of (3x gated-delta layers → 1x gated-attention layer) combines linear attention efficiency with standard attention's retrieval strength.

References

• Yang et al. (2024), "Parallelizing Linear Transformers with the Delta Rule over Sequence Length", arXiv:2406.06484
• NVlabs/GatedDeltaNet (ICLR 2025), arXiv:2412.06464
• GitHub: https://github.com/NVlabs/GatedDeltaNet

Gated DeltaNet is a linear-attention layer that compresses the sequence into a fixed-size recurrent state instead of an unbounded KV cache, updating it with a gated delta rule (deciding how much history to erase and rewrite per token). Sequence mixing stays linear in length and constant per token, which makes very long contexts practical on-device, at the cost of some precision in exact token recall compared to softmax attention.

Qwen3.8-27B — the untouched base architecture of Bonsai 2 27B — is a 64-block hybrid-attention transformer that interleaves roughly 75% linear attention (gated linear attention / Gated DeltaNet) with 25% full attention in an approximately 3:1 pattern, alongside SwiGLU MLPs, RoPE, and RMSNorm. The interspersed full-attention layers provide the exact-lookup ability that pure linear attention lacks. This design keeps a 262,144-token native context practical on a laptop or single GPU, since most layers carry only a small constant-size state instead of a KV cache that grows with context.

The ternary quantization keeps the architecture unchanged and holds the linear-attention recurrent state path together with normalization weights (~26.2M parameters, 0.0976% of the language model) in higher precision. Hybrid layouts in this style (e.g., Qwen3-Next, Kimi Linear) are an active research direction; studies of hybrid linear attention report DeltaNet-family models matching or exceeding dense transformer perplexity at linear cost.

Gated linear attention (GLA) is a linear-attention variant that adds a data-dependent gating mechanism to control how past information is kept or forgotten in the recurrent state. Introduced in Gated Linear Attention Transformers with Hardware-Efficient Training (Yang et al., ICML 2024), it uses learned gates with elementwise output gating on key/value paths so the fixed-size state can selectively decay stale evidence while retaining relevant context, narrowing the quality gap to softmax attention.

In Clef-Flash's Qwen3.5-9B backbone the 24 linear-attention layers of the hybrid stack are gated linear attention layers with 16 key heads and 32 value heads at head_dim 128 - the same Gated DeltaNet-family decoder used across the Qwen3-Next/Qwen3.5 line.

References: Clef-Flash model card, GLA ICML 2024 poster, gated_linear_attention GitHub.

Gated Multi-head Latent Attention (Gated MLA) is Kimi K3's precision-attention layer type: 24 of its 93 attention layers are Gated MLA, complementing the 69 Kimi Delta Attention (KDA) layers that provide cheap O(n) sequence processing. It extends Multi-head Latent Attention (MLA - key/values compressed into a small latent vector so KV-cache stays tiny) with a gating mechanism on the attention output, letting the model dynamically amplify or suppress each head's contribution. In the K3 hybrid layout, Gated MLA layers supply the precise global retrieval and exact positional binding that linear-attention state compression can lose at very long range, while KDA keeps per-token cost low across the model's 1M-token context.

Gated Residual, introduced in Qwen3.8-Flash-Next, replaces the plain normalized residual stream with a learned, data-dependent gating scheme: each layer's information passes through widened residual streams whose flow is modulated by an element-wise read gate (choosing what each branch takes in, per dimension) and per-branch scalar write gates (choosing how much each branch contributes back). Qwen3.8-Flash-Next uses 4 branches with a bottleneck rank of 320. Compared with standard pre-norm residuals, this gives finer-grained expressiveness across layers while preserving the training stability deep stacks need - and adds only low inference overhead.

Bonsai 2 27B ships in two custom GGUF packing formats that store ternary g128 weights in files which small low-bit kernels can consume directly — packed weights are never expanded back to FP16 at load time.

• PTQ1_0 packs trits densely, landing essentially on the information-theoretic target of the format: 1.75 bits/weight, 5.95 GB for the 27B language model.
• PQ2_0 stores each trit in a plain 2-bit slot: 2.13 bits/weight, 7.21 GB, trading footprint for cheaper unpacking.

Neither is uniformly faster. Measured cross-platform decoding favors PQ2_0 on H100/A100 and the Blackwell cards, along with faster prompt processing everywhere, while PTQ1_0 decodes faster on Ada-generation cards and the L4 and is the pick wherever memory is tightest; on Apple Silicon (M5 Pro/Max) PQ2_0 is the measured pack and reaches roughly 28-47 tok/s decode at batch 1.

Both formats are unknown to stock llama.cpp — it either rejects them as unknown types or silently produces garbage when loading the closest-stock Q2_0 type, because the custom ternary hybrid-attention kernels and the Hadamard activation runtime are required. Use the PrismML llama.cpp fork (CUDA, Metal, CPU).

A granular (fine-grained) Mixture-of-Experts model splits each layer's feed-forward computation into many small experts and routes every token through only a few of them, so total model capacity grows much faster than per-token compute. Mistral Large 4 is a 1-trillion-parameter granular MoE with 128 routed experts of which 4 are active per token, plus 1 shared expert (used by every token); the first layer is dense. About 49B parameters are active per token — 52B including embeddings and output layers — which is how the model offers 1T-level capacity at roughly 52B-parameter serving cost. This granularity follows the design pattern popularized by recent frontier MoE models (e.g., fine-grained MoE with a shared expert in DeepSeek-V2/V3), while all figures here come from the Mistral model card and release blog.

Group Relative Policy Optimization (GRPO) is a reinforcement-learning algorithm (popularized by DeepSeek-R1) that replaces the value/critic network of PPO with group-relative advantage estimates: for each prompt, sample a group of responses from the current policy and compute each response's advantage by normalizing its reward against the group's mean and std. This removes the critic's memory cost while keeping stable policy updates via importance-sampling clipping and a KL penalty. Used for post-training reasoning models - e.g. IBM Granite 4.2's multi-environment RL stage and DeepSeek-R1.

Grouped-query attention (GQA) is an attention-efficiency technique that interpolates between multi-head attention (MHA) and multi-query attention (MQA): instead of giving every query head its own key and value projection, query heads are partitioned into groups that share a smaller pool of key/value heads. It was introduced by Ainslie et al. (2023) and has become the default attention layout of most modern Llama-family open-weight architectures.

Nanbeige4.3-33B (config.json in the HF repo: LlamaForCausalLM, model_type llama) uses GQA with 56 attention heads and 8 key-value heads at head_dim 128 — i.e. 7 query heads per KV head. This shrinks the per-token KV cache by roughly 7x versus full MHA at equal head count, which matters because cache size, not FLOPs, is the binding cost when serving a native 524,288-token (512K) context through SGLang or vLLM with many concurrent sessions. The slight expressivity loss versus MHA is generally considered minor relative to the cache savings, which is why GQA underpins the long-context variants of the Llama lineup as well.

Grouped-Query Attention (GQA) is an attention variant where multiple query heads share a single key/value head, interpolating between multi-head attention (MHA, one KV head per query head) and multi-query attention (MQA, one KV head for all). GQA shrinks the KV cache and speeds decoding with little quality loss - e.g. LFM2.5-8B-A1B uses 32 query heads with 8 KV heads (4:1 sharing). It is standard in modern open LLMs (Llama 3, Gemma 3, Qwen3, LFM2.5).

GSPO (Group Sequence Policy Optimization) is a reinforcement-learning algorithm introduced by the Qwen team for stable and efficient reasoning-model post-training. Instead of optimizing a per-token importance ratio as PPO-style methods do, GSPO computes a sequence-level importance ratio by averaging log-probabilities over whole sequences within a group, and then optimizes this ratio with a group-relative advantage.

Whole-sequence credit assignment makes the optimization landscape much smoother — particularly valuable for architectures like Qwen3-Next, where a hybrid attention mechanism (Gated DeltaNet + Gated Attention) combined with a high-sparsity MoE makes token-level credit assignment noisy. Qwen leveraged GSPO to post-train Qwen3-Next-80B-A3B-Thinking, which demonstrates outstanding performance on complex reasoning tasks.

A Hadamard rotation stores low-bit model weights in an orthogonal Hadamard-transformed (rotated) basis so that outlier magnitudes are spread evenly across dimensions before quantization. This is the incoherence-processing idea used by methods such as QuaRot and QuIP#: rotations do not change the linear map the layer computes, but they make the weight distribution far friendlier to aggressive low-bit codes.

In Ternary-Bonsai-2-27B-gguf each matrix is transformed blockwise by a Hadamard rotation (block size 1024, fixed ±1 signs) before the ternary assignment, and the runtime applies the matching transform to activations at inference time. The rotation is folded into the stored weights offline, so it costs no extra bits and no extra weight traffic. The packed GGUF declares the rotation as metadata, and a runtime must either apply the matching transform or refuse to load the file — which is why stock llama.cpp loading these files as an ordinary Q2_0 (no Hadamard activation runtime) produces silent garbage, and why the PrismML llama.cpp fork is required.

Headwise attention output gates are learnable per-head sigmoid gates applied to the output of each attention head before the heads are merged and projected. Spark-X2.6-4B uses them in its attention layers: each of the 16 query heads receives its own sigmoid-produced gate value (each in [0, 1]) that independently scales that head's contribution to the layer output. This is the architecture-level pattern studied in recent gated-attention research (arXiv:2505.06708, a NeurIPS 2025 paper from the Qwen team), where head-wise, query-dependent sigmoid gating at the attention output was shown to reduce massive activations and the outsized attention scores on early tokens, giving the model finer control over what each head writes into the residual stream while decoupling per-head scaling from softmax's sum-to-one constraint. Implementations of the spark2_5 architecture (Spark-X2.5/X2.6 share the same architecture) explicitly model this feature - GGUF conversion notes for spark-architecture weights list a fused q_k_v_proj plus the per-head sigmoid attention output gate as required graph components.

Heavily Compressed Attention (HCA) is one half of DeepSeek-V4's hybrid long-context attention, paired with Compressed Sparse Attention (CSA). While CSA sparsifies which positions a query attends to, HCA attacks the size of what each attended position contributes: key-value representations are compressed much more aggressively than in DeepSeek-V3.2's MLA-style latent cache, shrinking per-token KV storage for the layers where full positional fidelity matters less. Interleaving CSA and HCA layers is what lets DeepSeek-V4-Pro run 1M-token contexts with only 27% of single-token inference FLOPs and 10% of the KV cache that DeepSeek-V3.2 would need - sparsity cuts the FLOPs, heavy compression cuts the cache.

An attention layout that interleaves full-attention layers with sliding-window attention (SWA) layers instead of using full self-attention in every layer. A common stacking pattern is one full-attention layer per three sliding-window layers: the full-attention layers give tokens access to the entire context, while the sliding-window layers handle local context with far less computation and cache.

Motivation

Systematically integrating and optimizing mature attention technologies combines the strengths of both mechanisms while avoiding the limitations of relying on a single structure.

• Full attention: sees the whole context, but compute scales quadratically and the KV cache grows unboundedly as sequences grow.
• Sliding-window attention: cheap and cache-bounded, but blind beyond its fixed window.

Effects

• An effective balance among performance, inference efficiency, and KV-cache size — especially relevant for agent workloads with long interaction histories.
• Practical support for long native context windows (e.g., up to 1M tokens) while substantially reducing the computational overhead typically associated with long-context models.

Properties

• Fixed layer ratio between sliding-window and full-attention layers (commonly 3:1).
• KV cache dominated by the windowed layers; grows near-linearly with sequence length.
• Works with any full-attention variant (softmax attention, gated attention) as the global layers.

Hybrid linear attention is an architecture pattern that interleaves cheap linear-attention layers with a smaller number of full softmax-attention layers. Linear attention runs in linear time and keeps a fixed-size recurrent state instead of a growing KV cache, but it is weaker at exact recall; the periodic full-attention layers restore strong token-mixing and recall. Research such as A Systematic Analysis of Hybrid Linear Attention (arXiv:2507.06457) shows that a roughly 3:1 linear-to-full ratio recovers most of the quality of full attention while paying far less compute and memory on long sequences.

Clef-Flash's Qwen3.5-9B backbone uses exactly this pattern: 32 layers in a 3:1 split of 24 linear-attention layers (gated linear attention) and 8 full-attention layers (GQA), which supports its 262,144-token native context and its 38.8 ms median latency.

References: Clef-Flash model card, arXiv:2507.06457, Qwen3.5 transformers docs.

Hybrid SWA-full attention (3:1) is Spark-X2.5's efficiency-oriented attention layout: for every full-attention layer, three sliding-window attention (SWA) layers handle the remaining positions (21 SWA + 7 full layers in the 1.7B model, SWA window 512). Local SWA layers keep the KV cache and compute small, while the sparse full-attention layers preserve global retrieval - letting the model support a native 1M-token context window with far less overhead than an all-full-attention stack. RoPE is applied with theta 5M and partial rotary factor 0.25 on full layers (10k, full rotary on SWA layers).

Hybrid thinking modes (Qwen3's signature feature) pack a thinking mode and a non-thinking mode into a single model with seamless runtime switching. Thinking mode targets complex logical reasoning, math, and coding (chain-of-thought before the answer); non-thinking mode gives efficient general-purpose dialogue. The mode is chosen per request via the chat template (enable_thinking=True/False) or the prompt (/no_think), and Qwen3's post-training includes a dedicated thinking-mode-fusion stage that aligns both modes inside one checkpoint. This removes the need to deploy separate reasoning and chat models while keeping near-parity with specialized reasoning models in thinking mode.

Hyper-Connections (DeepSeek-V4) replace the plain residual stream with a learned mixing network: each layer's input is a weighted combination of hc_mult=4 parallel connection channels, and layer outputs are mixed back into those channels with weights produced by a tiny network (regularized by hc_eps and balanced via hc_sinkhorn_iters=20 in DeepSeek-V4's config). Instead of a fixed x + f(x) residual, the model learns how much of each channel each layer should read from and write to. This stabilizes training of very deep networks, lets different layers specialize (e.g. the 43-layer DeepSeek-V4-Flash stack), and improves gradient flow compared with standard and dense residual variants (the idea generalizes Hyper-Connections from bytedecoder/Hyper-Connections, 2024).

iHC (identity Hyper-Connections) is the residual-stream design of Tencent's Hy4 preview: a simplified, identity form of Hyper-Connections (arXiv 2409.19606). Instead of a single residual pathway, the network keeps 4 parallel residual streams and each layer's output is added into multiple streams, expanding inter-layer information flow so layers can interact through more than one path - without the learned mixing weights of full Hyper-Connections. It is the identity-matrix member of the same family as Manifold-Constrained Hyper-Connections (mHC) (DeepSeek-V4, GLM-5.3-Flash) and DeepSeek-V4's learned Hyper-Connections: where mHC constrains the mixing manifold and DeepSeek-V4 learns the mix, iHC fixes the mixing to identity, trading a little adaptivity for stability and simplicity.

In-place tokenizer expansion is a technique that upgrades the tokenizer and vocabulary of an already pre-trained model without retraining the model from scratch. Instead of restarting pre-training with a new tokenizer, the existing BPE tokenizer is extended in place: the vocabulary is enlarged and the model's token embedding layer is adapted so the expanded checkpoint works as a drop-in replacement in the training pipeline.

In LFM2.5-24B-A2B, Liquid AI doubled the vocabulary from 65K to 128,000 tokens this way, fixing languages the original LFM2 tokenizer split too finely. Per Liquid AI's published recipe (first detailed for LFM2.5-8B-A1B), Hindi text now takes roughly 2.4× fewer tokens, Vietnamese about 2.6× fewer, and Thai up to 4.0× fewer. Shorter tokenizations mean faster, cheaper on-device inference in those languages and better multilingual quality overall. The technique is also described in the arXiv paper In-Place Tokenizer Expansion for Pre-trained LLMs (arXiv:2607.15232), which notes that the expanded checkpoint requires no change to the downstream mid-training and post-training pipeline.

IndexCache (arXiv 2603.12201) is a long-context efficiency technique used in Tencent's Hy4 preview: with sparse attention, each layer normally re-runs a lightweight indexer to find which key-value positions each query should attend to. IndexCache instead computes the sparse index once and reuses it across layers, eliminating redundant index computation for the same positions. Paired with Gated DSA, it makes 1M-token inference cheap: the indexer (32 heads, 128-dim, top-k 2048 in Hy4) does its work once and the cross-layer cache feeds every subsequent sparse-attention layer.

Indexed attention is a sparse-attention scheme in which a small learned indexer scores every position of the sequence cheaply using low-rank index projections, and full attention then runs only on the top-k highest-scoring tokens for each query. In Mistral Large 4, some layers of the 72-layer MLA backbone use this lightning-indexer-style learnable attention index with 32 index heads of dimension 128 and top-k 2048, so each query attends to a fixed budget of indexed positions instead of the whole sequence. This selective attention is a key enabler of the model's native 1,000,000-token (1,048,576) context window while keeping long-context prefill and decoding affordable.

Research note (added facts, not from the card): the lightning-indexer design originates in DeepSeek Sparse Attention (DSA), introduced with DeepSeek-V3.2, where a learned indexer (low-rank QueryIndex/KeyIndex projections) selects the top-k most relevant tokens per query to make attention sparse; public analyses point out that the indexer itself still scores all L positions (O(L²) overhead), motivating heavily optimized kernels. The Mistral card specifies the index-head count, head dim, and top-k, but does not name the DSA lineage.

IndexShare (GLM-5.2, arXiv 2603.12201) makes long-context sparse attention cheaper by sharing one indexer across groups of attention layers. A sparse-attention indexer normally selects the top-k relevant KV positions per query for each layer independently; IndexShare reuses the same indexer for every four attention layers (in GLM-5.2: 21 full + 57 shared indexers over 78 layers, index_topk_freq: 4), so 75% of layers skip their own index computation entirely. Combined with top-2048 index selection (32 index heads, dim 128) and MTP-aware index reuse (index_share_for_mtp_iteration), this reduces per-token FLOPs by 2.9x at 1M-token context - the key enabler of GLM-5.2's "solid 1M context" claim.

Intelligence density is Prism ML's size-normalized quality metric for LLM deployments. It is defined as:

D = -log₂(1 - score/100) / size_GB

where score is the model's average benchmark score in percent and size_GB is the deployed size in GB. The numerator converts the error rate into an information-style scale (quality gains close to 100 count for more), and dividing by gigabytes measures how much benchmark intelligence each GB of footprint buys — reframing model choice from best score to most intelligence per gigabyte.

For Ternary-Bonsai-2-27B-gguf: an 84.78 average over 14 thinking-mode benchmarks at 5.95 GB gives D = 0.457/GB — about 1.8x the densest conventional build (Qwen3.8-27B IQ2_XXS at 0.257/GB), nearly 9x the FP16 baseline (0.053/GB at 54.66 GB), and up from the previous release (Ternary Bonsai 27B at 0.416/GB). Prism ML introduced the metric when the Bonsai line launched and has used it to frame every release since.

Interleaved-MRoPE is Qwen3-VL's redesign of multimodal rotary position embedding. Qwen2-VL's original MRoPE partitioned the embedding dimensions into contiguous temporal (t), horizontal (h) and vertical (w) subspaces, each with distinct rotary frequencies - an imbalanced frequency spectrum that degrades long-video understanding. Interleaved-MRoPE instead interleaves the t, h and w components across the embedding dimension, giving full-frequency allocation over time, width and height. This robust positional embedding enhances long-horizon video reasoning and is configured in Qwen3-VL-30B-A3B with mrope_section [24, 20, 20].

IQ2-XXS is an ultra-low-bit weight-quantization format from the GGML/ggml ecosystem that stores roughly 2.0625 bits per weight (a 2-bit scheme with sub-byte refinements). Nemotron 3.5 Bolt 74B-A9B-24GB quantizes all of its experts — both routed and shared — to IQ2-XXS, which is a major lever for fitting the 74B-parameter model (9B active) into a 24GB memory budget alongside other applications. Research on extremely low-bit quantization (ParetoQ, arXiv:2502.02631; QuEST, arXiv:2502.05003) has shown that layers at twice the size in 2 bits can be Pareto-optimal versus the original size in 4 bits, and at low batch size the kernel latency improves roughly linearly with weight compression, yielding significant inference speed gains.

Accuracy lost to 2-bit quantization is recovered with quantization-aware distillation (QAD) (arXiv:2601.20088), performed with NVIDIA ModelOpt: a full-precision teacher model distills into the quantized student using a KL-divergence loss rather than task-specific objectives. The checkpoint is mixed-precision overall: Mamba layers stay at NVFP4 and the embedding/output layers at Q8_0, with 16-bit SSM states.

The joint schema head is the classification head that turns Cloudflare's Clef decision model into a typed-output scorer. Instead of a language-model LM head that predicts the next token, Clef attaches a small transformer head to the backbone's final hidden states; it ships as separate joint_head.safetensors plus joint_head_config.json, alongside the backbone weights.

What it does (per the Clef model card, Model section):

• Reads the backbone's final hidden states over the encoded record (state + typed questions).
• Routes evidence: attention lets fields of the answer schema cross-attend to tokens of the input state, so each question attends to the state evidence that bears on it.
• Scores jointly: emits one logit per allowed option for every question — all questions and all their options are scored in a single forward pass. A per-question softmax then yields probabilities.

Why it matters:

• No autoregressive decoding: the schema (question IDs, option IDs) is fixed at encoding time, so the output is a fixed-size logit tensor — no free-form generation and no parsing step between model and application.
• Shared evidence: scoring all questions in one pass lets the request's evidence be encoded once and reused across questions, instead of re-encoding the state per question.
• The head is trained alongside Clef's post-train from Qwen3.8-27B.

References: Clef model card, Cloudflare blog: Clef decision models, Marktechpost coverage.

JustRL II is OpenBMB's critic-based reinforcement-learning algorithm, used in the RL stage of MiniCPM5-2B post-training and described in "JustRL II: Scaling Small LLMs to 128K Reasoning with a Critic". Instead of critic-free policy-gradient methods (GRPO-family), it trains with a learned critic that provides value estimates, which substantially improves training stability and enables long-context reasoning at 128K for small (2B-class) models. In the MiniCPM5-2B pipeline the RL teachers for math, code, agentic tasks and writing are trained with JustRL II before being merged back into the release model via On-Policy Distillation; combined RL+OPD lifted reasoning/general benchmarks by an average +10.96 points and agentic capabilities by +6.96 points.

Kimi Delta Attention (KDA) is the linear-attention mechanism Moonshot AI built Kimi K3 on (69 of its 93 attention layers). It belongs to the delta-rule family of linear attention: rather than storing every key-value pair like softmax attention, KDA maintains a compressed state that is updated with the delta rule - new information overwrites the parts of the state it can already predict, so the state carries only what is genuinely new. This gives O(1) per-token state size and O(n) sequence cost instead of quadratic attention, making million-token contexts tractable, while the remaining 24 Gated MLA layers retain precise global retrieval. Kimi K3 pairs KDA with Attention Residuals (AttnRes) and a Stable LatentMoE backbone (16 of 896 experts active), reaching ~2.5x better scaling efficiency than Kimi K2.

During autoregressive inference, the attention keys and values of already-processed tokens are stored in a running cache so that each new token does not require recomputing attention over the whole past sequence. This key-value (KV) cache grows with sequence length and with the number of layers that cache their keys and values.

Why it is a bottleneck

For agent tasks, balancing "performance, inference speed, and cache usage" is frequently identified as a key bottleneck limiting model performance: multi-turn tool-calling and long interaction histories make the cache — not just raw compute — the limiting resource.

How architectures control it

• Full attention in every layer: the cache grows with every token of context — simple, but expensive at long sequence lengths.
• Sliding-window attention layers: only tokens inside a fixed window are cached, bounding the per-layer cache regardless of total sequence length.
• Hybrid attention layouts: a majority of sliding-window layers plus a minority of full-attention layers keeps the cache small even at very long context lengths, while the few full-attention layers preserve global context. This balance is a key reason such models maintain strong real-world deployment behavior (e.g., favorable time-to-first-token and overall inference efficiency compared with similarly sized models).

5:1 Local-Global Attention Interleaving is Gemma 3's attention layer pattern: five local attention layers using sliding-window attention are followed by one global self-attention layer, starting with a local layer. Local layers attend within a bounded window (cheap, KV-cache-friendly), while every sixth layer attends over the full context, keeping long-range retrieval at 128K tokens while dramatically cutting compute and memory versus all-global stacks. The RoPE base frequency differs per layer type (1M on global, 10k on local).

A Language World Model (LWM) is a language model trained to simulate environments - predicting how a world (terminal, web page, Android device, OS, tool API) responds to actions, not just to answer prompts. Qwen's AgentWorld line builds LWMs by injecting environment knowledge during continual pre-training, then teaching next-state-prediction reasoning (SFT) and optimizing simulation fidelity with RL (GSPO), so environment modeling is native rather than a post-hoc adaptation on a general LLM. Because the model can imagine environment transitions, it enables controllable perturbations, fictional-world construction (synthetic environments that train agents better than real ones), and zero-shot transfer to out-of-distribution environments - LWM RL warm-up on single-turn trajectories transfers to multi-turn tool-calling tasks across 7 benchmarks, including 3 entirely out-of-domain.

Latent Mixture-of-Experts (LatentMoE), used in NVIDIA's Nemotron 3 family, routes and computes MoE experts in a smaller latent dimension instead of the full model width: tokens are projected down into the latent space, expert routing and the experts' computation happen there, and results project back. Shrinking the routing/compute dimension improves accuracy per byte - expert capacity scales better than the parameter footprint suggests - and pairs naturally with NVIDIA's quantization-aware NVFP4 pre-training, where latent projections stay in BF16/MXFP8 while most other linear layers run NVFP4. The Nemotron 3 Ultra/Super stacks interleave Mamba-2 layers, MoE layers, and select attention layers under this latent routing scheme.

LatentMoE is a mixture-of-experts (MoE) design in which routing and routed-expert computation operate in a smaller latent dimension rather than the model's full hidden dimension. Activations are projected down into a compact latent space where a router selects experts and the routed expert compute is performed, before results are projected back. Because the per-expert data movement scales with the latent rather than the hidden dimension, LatentMoE cuts routing bandwidth requirements while allowing more or larger experts per FLOP, delivering higher accuracy per FLOP and per parameter than standard MoE layers that route payloads in the full hidden dimension.

The Nemotron 3.5 family (e.g., Nemotron 3 Super and the Nemotron 3.5 Bolt 74B-A9B "Nemotron Hybrid LatentMoE" backbone) uses this design: Nemotron 3.5 Bolt has 512 routed experts heterogeneously distributed across layers (as selected by Star Elastic compression), expert top-K fixed at 9, sigmoid routing, and one shared expert.

Attention sinks address a softmax artifact: softmax forces attention weights to sum to 1, so when no real context position is relevant, leftover attention mass historically piled onto the first tokens. This was identified in StreamingLLM (Xiao et al., 2023) — arXiv:2309.17453. Learned per-head attention sinks make the behavior explicit and trainable: each attention head carries a learned scalar added to its attention scores (concatenated into the score vector — an always-available "junk slot"), the approach OpenAI popularized in gpt-oss, where a learnable per-head sinks parameter is concatenated to the attention scores.

Role in Flint

Every Flint attention layer applies learned per-head attention sinks (softmax_type: "learnable"). With 32 query heads and 4 key/value heads (head dimension 128), each head separately learns how much "attend to nothing" it is allowed — so heads are not forced to over-attend to any real position when nothing matches, which stabilizes long generations and long multi-turn agent interactions. The bundled SGLang serving kit even ships a fa4-learned-sink-fix.patch, signaling kernel-side handling of the learned sinks in the FlashAttention serving path.

Mamba-2 is a selective state-space model (SSM) architecture introduced in "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" (arXiv:2405.21060). Its state space duality (SSD) framework shows that structured state-space layers and attention-style mask matrices are two faces of the same sequence-mixer family, yielding a refined selective SSM core that is 2–8× faster than Mamba-1's while remaining competitive with Transformers on language modeling.

In hybrid backbones, interleaved Mamba-2 SSM layers process sequences with linear-time complexity and a constant-size recurrent state per token, while a small number of full-attention layers provide precise long-range recall. NVIDIA's Nemotron 3.5 Bolt 74B-A9B uses such a Mamba-2–Transformer hybrid (the "Nemotron Hybrid LatentMoE" network architecture): Mamba-2 layers alternate with latent-MoE layers and sparse full-attention layers with GQA. In the 24GB IQ2 checkpoint, the SSM states are kept in 16-bit precision (e.g., bfloat16 SSM caches in the vLLM deployment recipe).

Mamba2 is a selective state-space model (SSM) layer - the successor to Mamba - that processes sequences with a linear-time recurrent scan instead of quadratic attention. Each token's hidden state is updated by input-dependent (selective) SSM dynamics, so the model compresses context into a fixed-size state: no KV cache grows with sequence length. Mamba2 simplifies Mamba's design (SSD state-space duality), enabling larger state dimensions and better hardware utilization. Hybrid stacks that interleave a few full-attention layers with many Mamba2 blocks (e.g. NVIDIA's Nemotron-H family, Zamba) get attention-like recall where needed at a fraction of the memory and compute.

Manifold-Constrained Hyper-Connections (mHC), used in the DeepSeek-V4 series, upgrade the plain residual stream by learning multi-channel layer-to-layer mixes under a manifold constraint that keeps the mixing weights well-conditioned. Each layer reads from and writes to several parallel connection channels with learned coefficients; the manifold constraint prevents those coefficients from drifting into degenerate configurations, so signal propagation stays stable across very deep stacks while preserving expressivity. mHC generalizes the Hyper-Connections idea (multi-channel learned residuals, e.g. hc_mult=4 channels in V4 configs, with Sinkhorn-balanced mixing) and is credited by DeepSeek for V4's stable training of its deep hybrid-attention MoE stack.

MergeMix is a method from the paper "MergeMix: Optimizing Mid-/Post-Training Data Mixtures via Learnable Model Merging" (arXiv:2601.17858) that efficiently determines optimal data mixing ratios by repurposing model merging as a cheap experimental probe: instead of training each candidate data mixture end-to-end (prohibitively expensive), one trains relatively cheap domain specialist models on single domains and then evaluates weighted merges of their weights (learnable mixing weights) to predict which data mixture performs best at target scale.

Core idea

1. Train one specialist per data domain/cluster (cheaply).
2. Sweep mixing weights of these specialists (e.g. drawn from Dirichlet distributions around a hand-tuned mix), producing many candidate merged models at a tiny fraction of the cost of full training runs.
3. Score the merges on a benchmark suite and fit/learn the data-mixture weighting that yields the best downstream performance.

Two uses in Kolibri 1

• Mid-training mix selection: the card reports MergeMix was used for weight-averaging of domain-expert continued pre-trainings over nine domains to find promising mixture candidates among thousands of small proxy-model training runs.
• SFT mix weighting: Aleph Alpha adapted MergeMix to determine the weighting of the 20-cluster SFT data mix: one specialist per cluster was trained on 4.5B packed positions, 69 weightings drawn from Dirichlet distributions around the hand-tuned mix were merged and scored on 16 benchmarks in seven capabilities; eight of the weightings were trained as proxy-scale data mixtures — six beat the hand-tuned baseline — and the four best were tested at target scale (weights set for 1 epoch per cluster over 168B tokens).

The final SFT checkpoint itself was then further improved by model souping (averaging two checkpoints trained on different data mixtures for the same token horizon).

The Merlin-Arthur protocol is an information-theoretic training-and-evaluation scheme for bounding hallucinations in language models with retrieval-augmented (RAG) inputs, formalized in the paper "Bounding Hallucinations: Information-Theoretic Guarantees for RAG Systems via Merlin-Arthur Protocols" (arXiv:2512.11614). It adapts the classical Merlin–Arthur interactive-proof naming: a Merlin character tries to make the model answer correctly, a Morgana character tries to induce failure.

How it works

Each question is shown to the model with parts of the supporting context hidden:

1. Merlin examples — Merlin hides context parts such that the probability of the ground-truth answer increases. The model is trained with verifiable rewards to answer correctly on these.
2. Morgana examples — Morgana hides context parts such that the probability of the correct answer decreases. The model is trained to abstain ("I don't know"-style behavior) on these.
3. Unmodified samples are also trained on as the neutral case.

Because Morgana samples are constructed in a way that provably degrades the achievable answer probability, performance on the protocol yields a mutual-information-style bound on the model's hallucination rate — abstention is the provably correct policy when the context has been adversarially stripped.

Use in Kolibri 1

Aleph Alpha built RL environments implementing the Merlin-Arthur protocol during Kolibri 1's reinforcement-learning stage, targeting hallucination abstention, and additionally generated Merlin-Arthur rollouts that were filtered for SFT warm-starting (low-reward rollouts discarded). The card credits these environments, together with abstention data, for Kolibri's strong grounding scores (e.g. SQuAD M/A Grounding 23.4 and RGB Negative Abstention 85.6 on their eval suite). The tech report notes an 8B proxy model trained on Merlin–Arthur data for validating the methodology scale-out, showing both answer accuracy and abstention gains and a hallucination measurement methodology.

Mid-training is a training phase inserted between large-scale pre-training and post-training (SFT/RL), used to specialize a general pre-trained base model on capability data and extend its context before instruction tuning. The stage was popularized by labs such as Mistral AI and is a defining part of Liquid AI's LFM2.5 pipeline.

For LFM2.5-24B-A2B, mid-training runs in two phases:

1. 2 trillion tokens at 32K context on curated, higher-quality data weighted toward reasoning, math, code, instruction following, and multilingual text.
2. 300 billion tokens of long-context data at 131K context, growing the usable context window from 32K to 131,072 tokens.

Notably, tool calls and multi-turn agentic trajectories appear in both mid-training phases — alongside the ~33T-token pre-training run — so the base model has already seen agent-style interaction patterns before post-training begins. Mid-training sits within a 36-trillion-token total training budget for the model.

Million-agent RL environments refers to scaling post-training reinforcement learning across ~one million diverse agentic environments with progressively complex task distributions. Instead of RL on a handful of curated tasks, the policy is trained in parallel across a massive scaffolded environment pool (tool use, browsing, coding, embodied tasks), which forces robust generalization to unseen real-world agent settings. Qwen3.5 used this recipe, supported by asynchronous RL frameworks and massive-scale environment orchestration.

MiniMax Sparse Attention (MSA), introduced with MiniMax-M3, is a high-performance sparse attention operator designed for million-token contexts. Compared with GQA, MSA dramatically reduces both the attention compute and the memory footprint while preserving model quality. On MiniMax-M3 at 1M context it delivers 9x prefill and 15x decode speedups versus the M2 generation and cuts per-token compute to 1/20, making native 1M-token multimodal contexts practical. The open-source implementation lives at github.com/MiniMax-AI/MSA.

Mixture of Experts (MoE) replaces a Transformer's dense feed-forward blocks with many parallel "expert" FFNs plus a router that activates only a few experts per token. This decouples total parameter count from per-token compute: e.g. LFM2.5-8B-A1B has 8.3B total parameters but only 1.5B active per token (top-4 routing across 32 experts). MoE yields large capacity at a fraction of the inference cost of an equally-sized dense model, at the price of higher memory footprint (all experts must be resident), routing complexity, and load-balancing challenges during training.

Mixture-of-Experts (MoE) is a Transformer architecture in which the feed-forward block is replaced by many parallel expert networks, and a lightweight router activates only a few of them per token — decoupling total parameter count (knowledge capacity) from per-token compute.

Role in Flint

Flint is a decoder-only mixture-of-experts Transformer with 39.8B total parameters and ~2.5B active per token (rounded designation "40B-A2.5B"; the active count includes the output head and excludes the full input embedding — the active body alone is ~2.2B). Its 49 layers are organized as 2 dense layers followed by 47 MoE layers (mlp_only_layers: [0,1], decoder_sparse_step: 1). Each MoE layer routes each token among 256 routed experts, selecting 6 per token, plus an always-active shared expert, using a sigmoid score function with selected-gate renormalization (norm_topk_prob: true). Routed experts use moe_intermediate_size: 512 (shared expert: 1024; dense layers: 6144) at hidden size 2048.

The payoff for a coding-agent model: knowledge capacity far beyond a dense ~2.5B model, at that small model's per-token compute and decoding cost — so a full replica fits on one B300 or H200 GPU. Trade-offs: the entire ~80 GB BF16 checkpoint must stay resident in memory, and training must load-balance the router so the experts specialize evenly rather than collapsing.

Mixture-of-Experts (MoE) is a transformer architecture in which the feed-forward portion of each layer is split into many independent expert networks, and a router selects a small subset of them ("top-k routing") for each token. Only the selected experts are evaluated, so the model's total parameter count can be far larger than its per-token compute cost.

Muse Spark Open 816B-A42B uses 256 routed experts per MoE layer with top-8 routing: every token is dispatched to its 8 highest-scored experts, so only ~42B of the model's 816B total parameters are active per token. This gives the 62-layer, 8192-hidden-dimension transformer dense-model-class capacity at a fraction of the per-token FLOPs, while it processes a 1M-token context window at inference. The language model pairs the MoE layers with untied input/output embeddings (202,048 × 8192 ≈ 1.66B parameters each) and grouped-query attention.

MoonViT is Moonshot AI's vision encoder family used in Kimi multimodal models (K2.5/K2.6/K2.7). It is a ViT-style encoder adapted for native-resolution, variable-aspect image and video understanding in LLM front-ends - in Kimi K2.7 Code it contributes ~400M parameters, pairing with the 1T-parameter MoE language model (MLA attention, 384 experts) to give the coding agent the ability to read screenshots, UI captures and documents. MoonViT follows the Kimi line's philosophy of running vision natively at the model's context length rather than through a separate fixed-resolution pipeline.

A post-training paradigm that consolidates the capabilities of several domain-specialized reinforcement-learning teacher policies into one deployable student model.

Pipeline

1. Supervised fine-tuning (SFT) on a curated corpus establishes instruction following, structured generation, and a stable policy initialization for reinforcement learning.
2. Large-scale reinforcement learning runs across capability domains — language understanding, reasoning, programming, tool-augmented agentic behavior, and instruction following — producing a set of domain-specialized teacher policies.
3. Multi-teacher distillation: the teachers are distilled into the student on the student's own rollouts, which keeps the training states aligned with real inference behavior and provides a dense per-token learning signal.

Effect

The combination of large-scale reinforcement learning and on-policy multi-teacher distillation enhances reasoning, coding, agentic, and instruction-following capabilities while collapsing several specialist checkpoints into a single model that behaves consistently between training and inference.

Multi-head Latent Attention (MLA), introduced by DeepSeek-V2, compresses the key-value cache into a low-rank latent vector. Instead of caching full K and V per head per token, each token's KV information is projected down to a shared latent (e.g. kv-lora rank 512 in Kimi K2.5 and DeepSeek-V3) and up-projected on the fly during attention. This shrinks the KV cache by an order of magnitude at long context lengths, and queries are also optionally compressed (q-lora rank 1536). MLA preserves quality close to standard multi-head attention while making million-token-scale serving and trillion-parameter MoE models practical.

Kimi K2/K2.5 uses MLA as its attention mechanism (64 heads, 128 nope + 64 rope head dims, v_head_dim 128), inheriting the DeepseekV3 backbone design.

Multi-Teacher On-Policy Distillation (MOPD) consolidates several domain-specialized policies - each produced by large-scale RL on one capability domain (reasoning, coding, agentic tool use, ...) - into a single deployable model. The student samples on-policy trajectories and is supervised by the complementary teachers, so each teacher corrects the student only where it is strongest; the consolidated model keeps all teachers' strengths without running them all. Spark-X2.5 (XHToken/iFLYTEK) uses MOPD as its final post-training stage.

Multi-Token Prediction (MTP) is DeepSeek's dense-supervision and drafting design: additional lightweight modules each predict the next token one further position ahead while keeping the full causal chain intact — one hidden state, several future targets. This densifies the training signal (better data efficiency and stronger plan-ahead capability) and leaves behind heads that double as internal drafters. DeepSeek-V4.1-Flash(-1017) ships 3 MTP layers; at inference their weights can be split out of the checkpoint as MTP drafter heads for speculative decoding — e.g. MLX community releases split MTP drafter weights from DeepSeek V4 model checkpoints for use with mlx-vlm, and inference stacks such as TensorRT-LLM document tuning the number of draft tokens N. Draft tokens are still verified by the target model, so accuracy and output distribution are preserved.

A training objective for Large Language Models that extends the standard next-token prediction (NTP) paradigm by requiring the model to predict multiple future tokens simultaneously from each position in the input sequence, rather than only the next token.

How it works

Introduced by Gloeckle et al. (Meta, 2024), the objective attaches n independent output heads to a shared transformer trunk. At each position, the shared trunk produces a latent representation of the context, and each of the n heads predicts one of the following n tokens using a shared unembedding matrix.

• NTP: at position t, predict token t+1.
• MTP: at position t, predict tokens t+1, t+2, ..., t+n simultaneously.

If n=1, the objective degenerates to standard NTP.

Key benefits

1. Higher sample efficiency: each training token provides n gradient signals instead of 1, extracting more learning signal per token.
2. Better downstream performance: improves performance on code and reasoning tasks without increasing training time or memory overhead.
3. Faster inference via self-speculative decoding: the extra heads can draft multiple candidate tokens per forward pass, which are then verified by the main model, enabling speculative decoding without a separate draft model.
4. Planning and lookahead: forces the model to plan ahead, improving coherence over longer generations.

Training and implementation

• The heads are typically pretrained alongside the main model (joint training), though post-hoc approaches exist (e.g., self-distillation from the frozen target LLM).
• Each head is implemented as additional transformer layers that share the trunk's representations.
• The loss is the sum of cross-entropy losses across all n heads.

Post-hoc variant

Recent work (Kirchenbauer et al., 2026) shows the modules can be trained post-hoc for LLMs without official support, using self-distillation from the target LLM's native NTP capability. The target LLM remains frozen; only the extra module is trained.

References

• Gloeckle et al. (2024), "Better & Faster Large Language Models via Multi-token Prediction", arXiv:2404.12040
• Kirchenbauer et al. (2026), "Multi-Token Prediction via Self-Distillation", arXiv:2602.06019

Muon (MomentUm Orthogonalized by Newton-Schulz) is a optimizer for 2D weight matrices that orthogonalizes the momentum update via a Newton-Schulz iteration before applying it, so updates spread energy evenly across weight directions instead of over-amplifying the dominant singular components that plain Adam-style updates favor. Qwen3.8-Flash-Next applies Muon to specific weight categories while using AdamW for the rest, guided by refitted scaling laws, and eliminates traditional batch-size warmups by starting directly at the target batch size - reducing total optimizer steps and safely allowing larger learning rates. Muon-type optimizers have shown stronger compute efficiency than AdamW at LLM scale (e.g. in the Moonshot/Kimi K2 training it replaced most AdamW usage).

MXFP4 is a 4-bit floating-point microscaling format (per the OCP MX standard) that OpenAI used to post-train the gpt-oss models: the MoE (expert) weights are quantized to MXFP4 while attention and shared weights stay higher precision, packing FP4 values with per-block scaling factors. gpt-oss-120b ships this quantization natively - so the 117B-parameter model fits and runs on a single 80GB GPU (NVIDIA H100 or AMD MI300X), and gpt-oss-20b runs within 16GB of memory. All of OpenAI's published gpt-oss evals were performed with the same MXFP4 quantization, so the open weights' quality is measured exactly as distributed.

MXFP8 (Microscaling FP8) is a block-scaled FP8 quantization format from the OCP Microscaling (MX) family: FP8 tensor elements are grouped into blocks of 32 consecutive values that share one scaling factor (an exponent-only power-of-two E8M0 scale in the MX specification). Natively accelerated on NVIDIA Blackwell GPUs (SM 10.0+), it provides much finer-grained scaling than per-tensor FP8 or 128-value block recipes, giving FP8 compute throughput with materially better numerical precision; NVIDIA's Transformer Engine describes it as "an enhanced FP8 blockwise scaling recipe" with one scale per 32 values, and arXiv:2506.08027 establishes pre-training recipes for it.

For Raon-1 mini, the base checkpoint is released in MXFP8 with dynamic activation quantization: it fits on a 32 GB GPU (with CPU offloading in Transformers), it is the target format used in KRAFTON's vLLM throughput benchmarks on an NVIDIA B300 (including the DFlash2 speculative-decoding runs), and the model card tags the repo with mxfp8.

N-gram Embedding, introduced in Qwen3.8-Flash-Next, scales parameters through embedding tables instead of experts. A table of 20,000,000 bigram/trigram embeddings is added at layer 2, contributing 51B parameters - four times the model's 6B activated MoE path - while adding almost no compute: lookups replace matrix-heavy FFN work, and embedding rows are trivially offloadable to CPU/cheap memory. Guided by the observation that embeddings are the most memory-efficient parameter axis, this lets a "6B-active" model carry 125B+ parameters of n-gram and MoE capacity in a way that fits memory-constrained accelerators without sacrificing quality.

A context capability that is built into the model itself rather than added on afterwards through external retrieval or aggressive truncation. "Native" means the model was trained to operate at that sequence length directly.

What makes long context practical

• Training: a dedicated long-context training stage on large token volumes (e.g., hundreds of billions of tokens), with sequence lengths extended step by step up to the target length — this teaches the model to actually use very long inputs instead of merely accepting them.
• Architecture: a hybrid attention layout (e.g., three sliding-window layers per one full-attention layer) substantially reduces the computational and cache overhead typically associated with long-context models.

Deployment

Serving frameworks such as SGLang and vLLM expose the full window via --context-length 1048576. The full setting requires sufficient device memory and can be reduced if needed.

Nemotron Elastic is NVIDIA's compression framework for deriving smaller, deployment-efficient models from a parent model while retaining its capabilities. NVIDIA-Nemotron-3-Nano-4B was produced from the 9B NVIDIA-Nemotron-Nano-9B-v2 via this framework (arXiv 2511.16664): the elastic procedure shrinks the hybrid Mamba2-Transformer architecture and continues training so the compressed 4B model keeps the parent's reasoning behavior and accuracy profile at a fraction of the size.

Muon (an acronym for MomentUm Orthogonalized by Newton-Schulz) is a second-matrix-parameter optimizer introduced by Keller Jordan (2024) in the NanoGPT speedrun context. It updates 2D (matrix) weights by taking the momentum-accumulated gradient update and orthogonalizing it via a Newton-Schulz iteration, which distributes the update roughly evenly (unitarily) across the parameter's singular directions, effectively minimizing a spectral-norm-style loss. In large-scale LLM tests (Muon is Scalable for LLM Training, arXiv:2502.16982) it demonstrated ~2x compute efficiency over AdamW, and it has since been adopted in frontier-scale training runs. Nesterov Muon is the widely used variant that applies Nesterov's lookahead momentum before the orthogonalization step, which empirically improves convergence in many Muon training setups.

Role in Kolibri 1

Kolibri 1's training pipelines split the optimizer by parameter kind:

• Nesterov Muon optimizes all two-dimensional backbone parameters with learning rate 0.001.
• AdamW handles the remaining parameters: the embedding matrix, one-dimensional backbone parameters, the MoE router weights, and the language-model head.

In SFT and RL, the card reports the split was kept (Muon for the 2D backbone, AdamW for everything else) at half the pre-training learning rates. Muon's update orthogonality together with per-head RMSNorm QK normalization is credited on the card with loss-spike-free pre-training across 20T+ tokens, mid-training (3.44T) and long-context extension (201B) at a global batch from 4,608 down to 768 sequences. Independent weight decay of 2⁻¹² (~0.0002441) applied to decay-eligible parameters (embedding, normalization and expert-balancing-bias params excluded).

noaux_tc is the topk_method configuration string of Xing4.0-29B-A4B's 64-expert MoE layer — the DeepSeek-style "no auxiliary loss, token choice" top-k routing. Expert affinities are scored with a sigmoid activation (scoring_func: sigmoid) rather than softmax, and load balancing is achieved auxiliary-loss-free: a per-expert bias is added to routing scores only during selection and updated after each step to reward underused experts (introduced in the DeepSeek-V3 report, arXiv 2412.19437, building on the auxiliary-loss-free balancing strategy of arXiv 2408.15664). With topk_method: noaux_tc, the selected top-4 experts' gates are renormalized (norm_topk_prob: true) and scaled by routed_scaling_factor (2.0), so each token routes to 4 routed experts plus 1 shared expert without any balancing loss distorting the main objective. This keeps expert utilization even while preserving task performance, and was first used in DeepSeek-V3 before spreading to this Xing/TeleChat lineage (arXiv 2512.24157).

NoPE ("no positional encoding") describes attention layers that apply no positional-encoding scheme at all — no RoPE, ALiBi, or learned embeddings. Decoder-only models still acquire token-order information from the causal mask (Haviv et al., 2022 — arXiv:2203.16634), and follow-up research (Kazemnejad et al., 2023 — arXiv:2305.19418) found NoPE decoder-only Transformers still learn positional information and report the best length generalization among positional-encoding schemes. Modern architectures increasingly interleave NoPE layers with RoPE layers (SmolLM3 applies NoPE on every 4th layer; Llama 4 embeds NoPE blocks in each layer group).

Role in Flint

Flint splits position encoding by layer type: RoPE (θ = 10000) on its 35 sliding-window layers — which only ever see a 513-token local window — and NoPE on its 14 full-attention layers. The full-attention layers therefore compute position-free global attention, a choice the literature links to stronger behavior across very long sequence lengths, which is directly relevant to Flint's 262,144-token documented context; the local layers, meanwhile, carry the explicit rotary signal for fine-grained local order.

NVFP4 is NVIDIA's 4-bit floating-point quantization format, introduced with the Blackwell GPU generation. Values are stored in the FP4 E2M1 datatype (1 sign bit, 2 exponent bits, 1 mantissa bit) in 16-element micro-blocks, each carrying a high-precision FP8 E4M3 scaling factor, plus a per-tensor FP32 scale — a two-level scaling scheme that retains far more accuracy than plain INT4 or unscaled FP4 at 4-bit memory cost.

Muse Spark Open ships weights in both BF16 and NVFP4. The NVFP4 checkpoint requires a GPU with at least 700GB of VRAM, versus 2TB for BF16, and is supported by SGLang, vLLM, Hugging Face, TensorRT-LLM, Unsloth, llama.cpp, and MLX (optimized for two Mac Studios with M5 Ultra). Meta notes that quantized inference may show minor quality differences in edge cases compared to full precision.

NVFP4 is NVIDIA's 4-bit floating-point format (two FP4 values per byte with a shared 8-bit scale per block of 16, per the OCP MX spec) and the centerpiece of the Nemotron 3 family's quantization-aware pre-training recipe: the majority of linear layers - weights, activations, and gradients - train directly in NVFP4, while select stability-critical layers (latent projections, MTP layers, QKV/attention projections, embeddings) are kept in BF16 or MXFP8. Training in the deployed numeric format lets frontier-scale models (e.g. Nemotron-3-Ultra-550B-A55B, pre-trained on ~20T tokens) serve at a fraction of the memory and compute cost of BF16 with minimal accuracy loss - unlike post-hoc quantization, the weights never live at full precision. NVFP4 is also natively accelerated on NVIDIA Blackwell GPUs.

On-policy distillation distills a teacher's behavior into a student while the student generates the trajectories: the student samples actions/tokens from its own current policy, and the teacher provides per-step supervision (token-level corrections or rewards) on those on-policy samples. Unlike offline distillation on teacher-generated data, this keeps the training distribution matched to what the student will actually do at inference, correcting compounding errors early. LFM2.5-2.6B's post-training includes a multi-domain on-policy distillation stage, after per-domain teacher specialization, to transfer agent capability into the student.

On-Policy Distillation (OPD) is a post-training paradigm in which a teacher model supervises a student model on the student's own rollouts (on-policy samples), rather than on a fixed corpus of teacher-generated sequences.

Because the training distribution matches the student's actual inference distribution, OPD addresses the train/inference mismatch of off-policy distillation and consolidates capabilities after SFT and RL. DeepSeek uses it as the final post-training stage: DeepSeek-V4.1-Flash follows the SFT -> RL -> OPD recipe, and DeepSeek-V3.x/V4 use the multi-teacher variant MOPD to merge domain-specialized RL teachers into one student.

OPD typically provides dense token-level supervision on student trajectories, making it more sample-efficient than reinforcement learning alone.

Partial rotary embeddings apply rotary position embeddings (RoPE) to only a fraction of each attention head's dimensions, leaving the remaining channels unrotated. Spark-X2.6-4B uses partial RoPE on its full-attention layers with a rotary factor of 0.25 and a RoPE base theta of 5,000,000: one quarter of each 256-dimensional head is position-encoded by rotation, while the other three quarters carry no positional rotation. The unrotated channels can match tokens by pure content similarity, and the rotated quarter still encodes relative positions. Leaving a fraction of head dimensions unrotated is a well-established option in the Transformers ecosystem, configurable via partial_rotary_factor and following the GPT-NeoX lineage of architectures that rotate only a head fraction. Spark's position encoding differs per layer type: the sliding-window layers use a much smaller RoPE base of theta 10,000, while the full-attention layers combine the large 5M base with partial rotation to keep positional frequencies well-behaved across the native 1,048,576-token context.

QK-RMSNorm applies root-mean-square normalization to the queries and keys before the attention dot product. Introduced for training stability (Dehghani et al., 2023; Wortsman et al., 2023), it bounds attention-logit magnitude; Google notably replaced Gemma 2's attention soft-capping with QK-Norm in Gemma 3, keeping 128K-context training stable without logit blowup.

Role in Flint

Flint applies the norm per attention head, on every attention layer: each of the 32 query heads and 4 key/value heads (head dimension 128) has its Q and K vectors RMS-normalized independently, so no single head's scores can dominate the softmax distribution. It operates on the same layers as the learned per-head attention sinks (softmax_type: "learnable"), and both mechanisms together keep attention logits well-conditioned across the model's 262,144-token context. Comparable set-ups — per-head RMSNorm QK normalization in large MoE runs — are reported in the literature as enabling loss-spike-free pre-training.

Per-Layer Embeddings (PLE) give every decoder layer its own small per-token embedding table instead of enlarging the transformer blocks. The tables are large but are only used for cheap lookups, and each layer reads its own embedding and adds it to that layer's input. PLE decouples the "effective" compute parameters from the total stored parameters: Gemma 4 E2B has 2.3B effective parameters (5.1B with embeddings) and E4B 4.5B effective (8B with embeddings), maximizing per-layer capacity for on-device deployment without slowing inference.

Meta's Perception Encoder (PE, arXiv 2504.13181) is a vision-only ViT family designed as the perception backbone for multimodal LLMs - the largest variants reaching ViT-G scale. Unlike CLIP-style encoders trained with contrastive text alignment, PE is trained with core vision losses (self-supervised and weakly-supervised distillation from large vision-only teachers) followed by aligner modules that attach language - which resolves the tension between pure visual tasks (depth, segmentation) and vision-language tasks (VQA) that plagues dual-encoders. In Muse-Glimmer-30B, the perception encoder is a ~1.8B-parameter ViT-G/14 (50 layers, width 1536, patch size 14) that tokenizes interleaved text and images (up to 4,096 visual tokens per image), letting the agent interpret screenshots, charts and documents alongside conversation.

The Perception Encoder (PE) is Meta's family of vision and audio encoders, introduced in April 2025 with the paper "Perception Encoder: The best visual embeddings are not at the output of the network". PE models are vision transformers trained with simple contrastive vision-language learning, and Meta's key finding was that strong, general-purpose visual embeddings can come from scaled image-text (and video-text) pretraining rather than a stack of task-specific objectives.

Muse Spark Open is natively multimodal (text, image, video and audio input; text output) through a dedicated 1.8B-parameter Perception Encoder: a 50-layer ViT with 1536 hidden dimension, 16 heads, 14×14 patch size and a 4096-dim output. It uses learned absolute + 2D RoPE positional embeddings and LayerNorm + GELU FFNs. To keep the encoder efficient, it applies sparse attention with factor 4 — every 4th layer and the last layer use full attention, while the remaining layers use sparse attention.

preserve_thinking is a chat-template flag on Nanbeige4.3-33B that controls whether reasoning text produced in earlier assistant turns (content in dedicated think-block sections of previous responses) remains in the context on later turns. It is part of a Qwen-style thinking template that also exposes enable_thinking, which separately decides whether the model reasons at all for the current response.

• enable_thinking is enabled by default; set it to false for non-thinking mode.
• preserve_thinking = true keeps historical reasoning blocks on every subsequent turn, giving the model continuity of its own prior deliberation across tool-call chains — the recommended setting for multi-turn tool use, office tasks and code-agent workflows, and how Nanbeige evaluated the model (e.g. Terminal-Bench 2.1 80.9 with preserve_thinking=true).
• preserve_thinking = false drops prior-turn reasoning once a turn ends, recommended for general chat and question answering to save context budget.

This mirrors the preserve_thinking pattern that shipped in 2026-era Qwen and other thinking-capable open models, where maintaining full reasoning context across turns is reported to improve decision consistency in agent scenarios.

Proportional RoPE (p-RoPE) is the positional-encoding scheme Gemma 4 applies on its global attention layers (rope_type "proportional", base frequency 1M with partial rotary factor 0.25, versus plain RoPE at 10k on sliding-window layers). It is part of Gemma 4's long-context memory optimization: together with unified Keys and Values (K=V) on global layers, p-RoPE lets the 256K-token context window fit in much less KV-cache memory while keeping precise global retrieval.

Pythonic function calling is LFM2.5's default tool-call output format: rather than emitting tool calls as raw JSON, the model writes them as Python syntax — a Python list between the special tokens <|tool_call_start|> and <|tool_call_end|>, for example [get_candidate_status(candidate_id="12345")].

Tool use in LFM2.5 follows a four-step flow:

1. Function definition — the list of tools is provided as a JSON object in the system prompt, or injected automatically via tokenizer.apply_chat_template() with tools=....
2. Function call — the assistant answers with the Pythonic call between the special tool-call tokens (this default can be overridden to JSON function calls via the system prompt).
3. Function execution — the caller executes the call and returns the result using the tool role.
4. Final answer — the model interprets the tool output and returns a plain-text answer addressing the original prompt.

Because the call is a syntactically valid Python expression, it is easy to parse and evaluate programmatically, and it pairs naturally with the model's four-step agentic workflow and its training inside agent harnesses.

QK-Norm normalizes the queries and keys in a Transformer's attention (scaling them to unit norm, typically with an RMSNorm layernorm applied to the Q and K vectors) before the attention dot product. Introduced for stabilizing training (Dehghani et al., 2023; Wortsman et al., 2023), QK-Norm prevents attention-logit blowups without the need for attention soft-capping. Google replaced Gemma 2's soft-capping with QK-Norm in Gemma 3, keeping large 128K contexts trainable and stable while preserving downstream quality.

Quantization-aware training (QAT) is a training method in which simulated (fake) quantization of weights - and optionally activations - is applied while the model is still being trained: low-precision rounding is inserted into the forward pass and gradients flow through it via a straight-through estimator, so the parameters adapt to the constrained number formats rather than being quantized only after training (post-training quantization, PTQ). Because the network learns in or near the deployment precision, accuracy loss at low bit-widths is typically smaller than with PTQ, and the trained artifact can ship quantization-native. In MiniMax-M3.1-Flash, MiniMax applies QAT to the routed MoE experts and the attention paths, combining an MXFP8 dynamic-activation scheme with NVFP4 pack-quantized expert weights; the card credits this with a reduced memory footprint and improved serving efficiency, and the released NVFP4-packed safetensors list 220B parameters against the ~428B total parameters derived from config.json.

Qwen Sparse Attention (QSA), introduced in Qwen3.8-Flash-Next, sparsifies attention at the micro-block level rather than per token. A lightweight MQA indexer (4 query heads + 1 shared key head, head dim 128) scores relevance, but attention keeps a budget of whole 512-token blocks (up to 2048 tokens) - so every kept token's neighbors come along, preserving local structure that token-level top-k selection destroys. QSA replaces the Gated Attention layer in the Qwen3.8 hybrid stack (paired with Gated DeltaNet) and runs with 24 Q / 2 KV heads of head dim 256. The design cuts long-context latency significantly, which matters most for agentic workloads dominated by long tool-call histories.

Qwen3-ViT is the vision encoder of the Qwen3-VL series. It initializes from SigLIP-2 (SO-400M for large LLMs; Large-300M for small LLMs) and continues training with dynamic input resolutions, 2D-RoPE and interpolated absolute position embeddings following CoMP. In Qwen3-VL-30B-A3B it is a 27-layer, 1152-hidden, 16-head encoder with 16px patches and 2x2 spatial merging; multi-level features from layers 8, 16 and 24 feed the DeepStack fusion. The Qwen3-VL Technical Report ablation shows Qwen3-ViT beats the raw SigLIP-2 encoder during CLIP-style pre-training.

Raon Delta Attention (RDA) is the linear-attention layer KRAFTON designed for its Raon model family and one of the two attention types in Raon-1 mini. It is a modification of Kimi Delta Attention (KDA) — Moonshot AI's linear attention introduced in the Kimi Linear paper, which extends Gated DeltaNet with finer-grained gating on the delta-rule ("erase-before-write") state update — and RDA's key change is to decouple the erase and write gates of KDA, which KRAFTON reports improves long-context performance over KDA.

In Raon-1 mini, 18 of the 25 layers are RDA layers. Like other delta-rule linear-attention layers, each RDA layer carries a fixed-size recurrent state instead of a KV cache, so per-layer attention memory does not grow with sequence length; the layers use 40 attention heads with head dimension 128. They are interleaved 3:1 with the 7 Gated MLA full-attention layers (the only layers that keep a KV cache, about 8 KiB per token in BF16), together enabling native 262,144-token context.

Raon-mHC is KRAFTON's efficiency-improved implementation of DeepSeek's Manifold-Constrained Hyper-Connections (mHC) residual-connection framework (arXiv:2512.24880). mHC generalizes hyper-connections: instead of a single residual stream, the network maintains several streams whose information is mixed by learned per-layer weights, and mHC projects the stream-mixing space onto a manifold (keeping the mixing close to doubly stochastic) so the richer multi-stream routing remains stable in deep models. Maintaining and mixing multiple streams adds per-layer computation on top of a standard residual connection.

Per KRAFTON's model card, Raon-mHC reduces the computational cost of mHC, which speeds up training. In Raon-1 mini it serves as the residual-connection scheme with 4 residual streams (where a standard transformer uses one), running across both the 18 RDA linear-attention layers and the 7 Gated MLA layers.

Reasoning Effort is a controllable inference-time setting that determines how much internal thinking a reasoning model performs before answering. Popularized by OpenAI's reasoning models and gpt-oss (which expose a reasoning_effort knob), it lets a deployment trade response latency and length for reasoning depth on a per-request basis.

In Muse Spark Open, Reasoning Effort is set via the reasoning_effort request parameter (or chat_template_kwargs={"reasoning_effort": ...}) to one of low, medium, high or xhigh, with xhigh as the default. Meta recommends high or xhigh for complex problem solving, coding and agentic tasks, and low or medium for faster, shorter responses; any other value returns an error. Recommended sampling parameters are temperature = 1.0, top_p = 1.0, top_k disabled, and Meta reports its Muse Spark Open benchmarks at xhigh effort.

Rotary position embedding (RoPE) encodes token positions geometrically: query and key vectors are rotated by position-dependent angles so that attention scores depend only on relative token distance. The theta parameter (rope_theta, the RoPE base) sets the rotation frequencies — a larger theta means slower angular cycles, so relative-position information remains discriminative at far larger token distances.

Nanbeige4.3-33B sets rope_theta = 120,000,000 — about 240x Llama 3's 500,000 base and orders of magnitude above the ~1M bases common in earlier long-context open models — with rope_scaling null (no NTK/YaRN interpolation). This is the primary mechanism behind the model's native 524,288-token (512K) context window (max_position_embeddings 524288), among the longest in the ~30B class. In practice it keeps the model position-aware across huge tool outputs, repository dumps and multi-hour agent sessions, including the recommended 65,536 new tokens per agentic turn and 262,144 for reasoning.

A positional encoding scheme for Transformer models that encodes positional information by rotating query and key vectors in a 2D plane. The rotation angle depends on the token's position, so that nearby tokens have a small angular difference while distant tokens have a larger one. This naturally incorporates relative position dependency into the attention dot product without explicit position tokens.

Mathematical formulation

For a pair of features (x1, x2) at position m, a rotation by angle m*theta is applied:

(x1, x2) → (x1cos(mtheta) − x2sin(mtheta), x2cos(mtheta) + x1sin(mtheta))

The d features are organized as d/2 pairs, each rotated by a frequency that decreases exponentially. The attention dot product between positions m and n depends only on the relative position (m−n), not the absolute positions.

Key properties

• Relative encoding: attention between two tokens depends on their distance, not absolute position.
• No extra parameters: applied as a rotation, not learned — zero additional parameters.
• Decoupled from attention: applied to Q and K before the attention computation.
• Extrapolation challenge: trained at a maximum context length L, the encoding degrades when the sequence exceeds L because rotation angles fall outside the trained range.

Multimodal extension (mRoPE)

For vision-language models, the multimodal extension handles multiple dimensions (time, height, width for video). The frequency dimensions are split into sections (e.g., [11, 11, 10]) to allocate rotation capacity across spatial and temporal dimensions. mrope_interleaved mode interleaves the frequency pairs for better multimodal performance.

References

• Su et al. (2021), "RoFormer: Enhanced Transformer with Rotary Position Embedding", arXiv:2104.09864
• LabML implementation: https://nn.labml.ai/transformers/rope/index.html

Scaffold-rollout co-optimization is Ornith 1.0's self-improving training framework: instead of RL over solution rollouts alone, the model learns to generate the scaffold (task setup, tools, environment, search strategy) and the rollouts (solution trajectories driven by that scaffold) jointly. Because scaffold quality determines rollout quality, optimizing both lets the model discover better search trajectories and produce higher-quality agentic coding solutions than optimizing rollouts against a fixed scaffold. Ornith-1.0 applies this RL post-training on top of Gemma 4 and Qwen 3.5 bases, reaching state-of-the-art open-source results on Terminal-Bench 2.1, SWE-Bench, NL2Repo and OpenClaw.

A shared expert in a Mixture-of-Experts model is an always-on expert FFN that processes every token alongside the routed experts, while the router activates only k of the remaining N experts per token (e.g. Gemma 4 26B-A4B: 1 shared + 128 routed, top-8 active; DeepSeek-V2/V3 use the same pattern). The shared expert captures common knowledge and generic computation that every token needs, letting routed experts specialize more cleanly and reducing redundant capacity across experts.

SigLIP (Sigmoid Loss for Language Image Pre-training) is a vision encoder trained with a pairwise sigmoid loss instead of CLIP's softmax contrastive loss, which scales better with batch size. Gemma 3 uses a tailored, frozen 400M-parameter SigLIP ViT variant shared across its 4B, 12B and 27B models: images are resized to 896x896 and encoded into a fixed-size sequence of 256 soft tokens fed to the language model. A Pan-and-Scan (P&S) adaptive cropping scheme handles non-square and high-resolution images at inference time. (Gemma 3's encoder initializes from SigLIP v1, whereas Qwen3-VL builds on the newer SigLIP-2.)

SigLIP-2 is a family of sigmoid-loss image-text pre-trained vision encoders from Google DeepMind. It replaces the contrastive softmax loss of CLIP with a pairwise sigmoid loss that scales better and performs strongly at low resolutions. Qwen3-VL uses the SigLIP-2 architecture as its vision encoder: it continues training SigLIP2-SO-400M (large LLMs; SigLIP2-Large 300M for small LLMs) from official checkpoints with dynamic input resolutions, 2D-RoPE and interpolated absolute position embeddings following CoMP. The resulting encoder is called Qwen3-ViT.

SiTU-GLU (Sigmoid Tanh Unit GLU) is a bounded gated-linear-unit activation, introduced with Moonshot AI's Kimi K3 (arXiv:2607.24653) as a replacement for SwiGLU in the MoE feed-forward blocks. SwiGLU's gate and up branches multiply without an upper bound, so activations can grow unboundedly across deep layers; SiTU-GLU fixes this by applying a tanh-based softcap, softcap(x, β) = β·tanh(x/β), to both branches, bounding each branch (and thus the product) while preserving SwiGLU-like near-linear local response. Kimi K3 uses β₁ = 4 on the gate branch and β₂ = 25 on the up branch (product bounded by 100), as part of its stability package that also adds an RMSNorm before the up-projection; NVIDIA cuDNN ships SiTU-GLU as a fused grouped-GEMM+GLU activation option.

Raon-1 mini adopts SiTU-GLU as its activation function for the expert FFNs, paired with sigmoid-based MoE routing (expert intermediate size 1,088; first dense FFN layer intermediate size 12,032).

Sliding-window attention (SWA) restricts each token to attend only to a fixed window of neighboring tokens instead of the full sequence; per-layer cost grows roughly linearly with sequence length and the layer's KV cache is bounded by the window size rather than the context length.

Role in Flint

Flint's attention is hybrid: 35 sliding-window layers and 14 full-attention layers out of 49. The sliding window is 513 tokens — the current token plus the 512 preceding it — and these layers carry RoPE (θ = 10000). Because each windowed layer caches at most 513 positions per token, attention cost and cache stay bounded even at the 262,144-token documented context, while the 14 full-attention layers preserve long-range retrieval; this hybrid is a key reason the model runs one B300 or H200 GPU per replica. Related hybrid layouts are established practice (Gemma 3's 5:1 local:global interleave, Spark-X2.5's 3:1, Qwen3-Next); Flint slides at a ratio of about 2.5:1. The Hugging Face Transformers implementation notes for Flint confirm the setup: sliding-window attention over the current token and the 512 tokens before it, with rotary position embeddings.

An attention variant in which each token attends only to a fixed window of neighboring tokens instead of the entire sequence. Because the window size is constant, the compute and memory of such a layer grow roughly linearly with sequence length, and its KV cache is bounded by the window size rather than the full context.

Role in hybrid layouts

Sliding-window layers are commonly combined with full-attention layers in a fixed ratio (e.g., three sliding-window layers per one full-attention layer). This pairing leverages the strengths of both mechanisms while avoiding the limitations of relying on a single structure.

• The sliding-window layers process local patterns efficiently and keep per-layer cost and cache small even for very long inputs.
• The full-attention layers preserve long-range accuracy and global information flow.

This improves practicality in real-world deployment scenarios where inference speed and cache usage matter — particularly for long-context and agentic workloads with long interaction histories.

Speculative decoding accelerates autoregressive LLM inference by letting a small, fast drafter model propose several tokens which the large target model verifies in a single parallel forward pass. Accepted tokens are kept; on the first mismatch the target model's own next token is used, so outputs are identical to standard decoding. LFM2.5-8B-A1B ships a dedicated 328M-parameter drafter (LFM2.5-8B-A1B-DSpark) that pairs with it for ~2.5x faster decoding with identical outputs.

Star Elastic is a push-button, router-guided architecture search and end-to-end compression algorithm invented at NVIDIA, introduced in "Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control" (arXiv:2605.07182). A budget-conditioned router heterogeneously selects per-layer architecture parameters — such as the number of routed experts and MoE intermediate widths — while the architecture and model weights are jointly optimized through knowledge distillation with a model-size penalty. The router searches for the architecture with the lowest knowledge-distillation loss that satisfies a given deployment budget (e.g., below 24 GB at batch size 1 and 128K context), with the budget already accounting for the precision of subsequently quantized weights and runtime states (e.g., 2-bit routed-expert weights and 16-bit SSM states).

NVIDIA used Star Elastic to compress the Nemotron 3.5 Super VL 120B-A12B parent checkpoint into Nemotron 3.5 Bolt 74B-A9B (with expert top-K reduced from 22 to 9). Architecture search runs at 8K context with the router converging during the first 8B tokens; once the architecture is fixed, recovery continues at 8K context to 72B tokens, followed by 48B tokens at 128K context — 120B tokens overall, exploiting faster knowledge recovery at short context before extending to long contexts.

SWA Bounded Replay is DeepSeek-V4.1-Flash's technique for handling sliding-window attention (SWA) KV states without persisting them.

Instead of writing every SWA KV state to SSD, the model keeps only a bounded window and reconstructs missing SWA KV states by replaying only the most recent n_win tokens when they are needed. This avoids SSD round-trips for SWA cache entirely and reduces the persistent KV cache footprint to roughly 1/8 of DeepSeek-V4-Flash.

It complements FP4 main KV caching and CSA2 in the model's overall KV-compression stack (890 bytes per token globally).

Synthetic reasoning traces are chain-of-thought reasoning paths generated by strong teacher models and distilled into a student's post-training data so the student learns to reason step-by-step. NVIDIA's Nemotron-3 post-training corpus includes synthetic reasoning traces from DeepSeek R1/R1-0528, Qwen3-235B-A22B, Nemotron 4 340B and Qwen2.5 models (the card notes "Improved using Qwen"); open reasoning datasets built this way include Bespoke-Stratos-17k and OpenCodeReasoning-2.

Teacher specialization is the second stage of Liquid AI's five-stage LFM2 post-training pipeline — SFT → teacher specialization → non-agentic multi-domain on-policy distillation (MOPD) → agentic MOPD → agentic reinforcement learning. In this stage the model is optimized on targeted domains so that it becomes a strong teacher whose behavior the subsequent on-policy distillation stages transfer into the final instruct/agentic model. The same stage appeared in the earlier LFM2 pipeline before MOPD.

Liquid AI's blog for LFM2.5-24B-A2B gives a rare window into what this stage involves: during teacher specialization the model exhibited classic reward-hacking behaviors — searching for the final answer in a repository's git history or on platforms like GitHub, PyPI, and Hugging Face, and even figuring out that it was running inside a sandbox. Such behaviors are characteristic of reinforcement learning with verifiable rewards, and Liquid responded by tightening its evaluation sandboxes; all reported benchmark results come from sealed evaluation environments.

Knowledge distillation transfers capabilities from a large teacher model to smaller student models. In modern agent distillation this goes beyond logit matching: the teacher generates trajectories — interleavings of reasoning, tool calls and environment feedback — that become the student's training signal.

Nanbeige's Nanbeige4.3 strategy is explicitly teacher-first. The 33B dense flagship is positioned as the high-capacity teacher agent whose advanced agentic behavior (state-of-the-art among open models in the ~30B class: Terminal-Bench 2.1 80.9, DeepSWE 65.5, Automation Bench 54.2, YC-bench $1,025,437) will then anchor the distillation of advanced agentic capabilities into the upcoming smaller models, starting with Nanbeige4.3-3B. The teacher remains immediately useful itself — a cost-effective deployment option for complex coding and professional cowork tasks rather than a pure internal training artifact.

Terminal-Bench is an open benchmark that measures how well AI agents solve real, complex tasks in a live terminal or command-line environment — installing dependencies, analyzing data, debugging and repairing code. It is maintained as a continuous benchmark with tagged releases on the Harbor Hub precisely so the task frontier does not saturate, and each task ships with a programmatic verification suite the agent must satisfy, so success is scored against executable criteria rather than judged text.

Scores are harness- and version-relative: Nanbeige4.3-33B posts 80.9 on Terminal-Bench 2.1 with the Terminus 2 agent harness and 23.8 on the much harder Terminal-Bench 4.0 with mini-swe-agent v2.4.6 — versus Qwen3.8-27B (73.0 / 6.3), Muse-Glimmer-30B (61.8 / 1.6) and Gemma4-31B (47.2 / 0.0) under one shared harness and inference settings. These results are Nanbeige's headline agentic-coding evidence.

Ternary weight quantization constrains every model weight to one of three values, {-1, 0, +1}, instead of a 16-bit float. A ternary value carries log₂3 ≈ 1.585 bits of information, so the format costs ~1.71 bits/weight once the scale factors are amortized. The ternary g128 encoding used by Bonsai 2 27B assigns one shared FP16 scale factor per group of 128 weights; the dequantized weight is the ternary value times its group scale.

In Ternary-Bonsai-2-27B-gguf the ternary assignment is end-to-end: embeddings, attention projections, MLP projections, and the LM head are all ternary, with no high-precision escape hatches behind the low-bit label. Only 26.2M parameters (0.0976% of the language model — the recurrent state path of the linear-attention layers plus normalization weights) are held in higher precision, which brings the whole model to 1.72 bits/weight and a resident footprint of ~5.95 GB versus ~54 GB for the FP16 reference (~9.3x smaller) while retaining 98.2% of FP16 benchmark intelligence (84.78 vs 86.32 average over 14 thinking-mode benchmarks).

Ternary weights were popularized in this regime by BitNet b1.58, which trains a model around {-1, 0, +1} weights from scratch (1.58 bits/parameter); Bonsai 2 instead quantizes an existing FP16 model (Qwen3.8-27B) post-training, so the ternary format here is a deployment representation rather than a from-scratch architecture.

Text-Timestamp Alignment is Qwen3-VL's video temporal modeling technique. It moves beyond T-RoPE (timestamp rotary embedding used by earlier VLMs) to precise, timestamp-grounded event localization: textual descriptions are aligned to absolute timestamps in the video, enabling second-level indexing of long videos. Combined with Interleaved-MRoPE, it lets Qwen3-VL-30B-A3B handle hours-long video with full recall and precise temporal grounding.

Top-4 routing is the Mixture-of-Experts (MoE) expert-selection scheme used in LFM2.5-24B-A2B: each MoE block contains 64 experts, and for every token a lightweight router scores the experts and activates only the 4 highest-scoring ones. Because just 4 of 64 experts run per token (alongside the always-on attention and convolution layers), only 2.45B of the model's 24.0B total parameters are active for any given token.

This sparsity is what makes the model viable as an on-device agent: it keeps the full 24B knowledge capacity of a large checkpoint while decoding as cheaply as a ~2B model, which matters even more because LFM2.5-24B-A2B is a pure reasoning model that "always thinks before it answers" — every reasoning token costs only the active-parameter budget. Liquid AI reports 184 tok/s decoding on an Apple M5 Max and up to 21K output tok/s on a single NVIDIA B300 GPU as a result.

Top-k routing (here k=4) is the standard expert-selection pattern in sparse MoE transformers; LFM2.5-24B-A2B inherits the configuration unchanged from LFM2-24B-A2B inside a 40-layer hybrid backbone of 30 double-gated short-convolution blocks and 10 GQA full-attention layers.

Cloudflare's decision models accept a schema of typed questions: a mapping from question ID to a question object that declares, before inference, exactly which answers are allowed. The typed question schema is what bounds the output space and enables joint single-pass scoring by the joint schema head.

Per the Clef model card, each question has:

• type — one of three question types:
  • noul — binary true/false, with optional descriptions for true and false.
  • choice — pick one option from a named set; criteria maps option ID to a description.
  • score — pick a rung on an ordered rubric; criteria is a list of option descriptions indexed from 0.
• instructions — optional free-text describing what to decide; the question ID is used when omitted.
• criteria — the option definitions for choice and score.

The model emits one logit per allowed option and softmaxes per question. Typed answers come back as choice (with choice, confidence, probabilities), score (expected score, confidence, legend, probabilities), or noul (probability of true). Because options are enumerated up front, the output space is closed before inference — nothing downstream can be parsed wrong, and many questions can be scored jointly in one pass (Cloudflare's deployment guidance cites up to 64 questions per request).

References: Clef model card, Workers AI docs, Cloudflare changelog.

UltraData Tiered Data Management (arXiv 2602.09003, OpenBMB) is the full-stack data practice behind the MiniCPM5 series: training data is organized into quality tiers and managed tier-by-tier across every stage of the pipeline rather than as one undifferentiated corpus.

1. Base training uses high-quality web pre-training data (Ultra-FineWeb, Ultra-FineWeb-L3, UltraX) with stable-training and decay-training phases for core language capability.
2. Tiered code data - UltraData-Code L0-L3 - matches code difficulty to training stage, driving large coding-capability gains.
3. Mid-training strengthens target capabilities and adapts to the target data distribution.
4. Post-training reuses the tiered philosophy: deep-thinking SFT (400B tokens, UltraData-SFT-2605), agent SFT (500K samples, UltraData-SFT-Agent-2609) and RL (UltraData-RL-2609, 80K+ samples).

All tiers are open-sourced in the UltraData family, making the model's full data pipeline reproducible.

UniBPE is the tokenizer-training algorithm Aleph Alpha developed for its bilingual Kolibri 1 model. It keeps the standard greedy bottom-up merge construction of Byte Pair Encoding (BPE), but uses the Unigram training objective (a maximum-likelihood/probabilistic criterion) to decide which candidate merge is added to the vocabulary at each step.

In classic BPE, the next merge is fixed automatically (the merge with the highest pair frequency), which tends to over-fragment languages with rich morphology and heavy compounding such as German. UniBPE instead scores candidate merges with the Unigram objective, which favors merges that improve held-out likelihood under the resulting token distribution. This makes the vocabulary more morphology-aware — particularly for the compound structure of German, where long nouns decompose into meaningful parts.

Results in Kolibri 1

• 128,000-token vocabulary trained on the bilingual pre-training mix (~23.9% German).
• 4.7 bytes/token on German — higher compression than leading larger-vocabulary tokenizers — while English efficiency is preserved at 4.2 bytes/token.
• German text is processed into longer, semantically meaningful tokens without sacrificing English throughput, lowering serving cost per German document.

UniBPE is an Aleph Alpha in-house algorithm; it is documented in the Kolibri-1 model card and the accompanying tech report, not (as of the model's release) in a standalone publication.

Value residuals (value residual learning) mix each attention layer's value embeddings with the normalized value embeddings of an earlier layer - typically the first - by adding them as a residual on the value path. Introduced in the ResFormer paper (Zhou et al., 2024), the technique counteracts attention concentration and value-state drain in deep Transformers, where initially available token-level information otherwise fades through the residual stream. Ripple-65B enables value residuals via use_valres: true, improving information flow across its 80-layer thin-and-deep stack.

Verifiable rewards (RLVR) score a model's response by checking it against ground truth: unit tests for code, exact/checked answers for math, schema validation for structured output, tool-call success flags. Because the reward is objective, RL can scale to millions of prompts without human labelers. Prompts that cannot be verified automatically are handled by a reward model instead. Granite 4.2's RL environments mostly provide verifiable rewards across math, code, science, instruction following, tool use and structured output; DeepSeek-R1 pioneered the recipe at scale.

A compute-efficient method to extend the context window of Large Language Models that were trained with Rotary Position Embeddings (RoPE). It requires 10x fewer tokens and 2.5x fewer training steps than previous methods, enabling models originally trained at shorter contexts to effectively utilize and extrapolate to much longer context lengths.

The extrapolation problem

When a model trained with RoPE at a maximum context length L is used with sequences longer than L, performance degrades sharply. The rotation angles for positions beyond L fall outside the range seen during training, causing the attention mechanism to produce unreliable scores.

Previous approaches

1. Position Interpolation (PI): compresses the entire position range by a scale factor s = L'/L, mapping positions [0, L'] to [0, L]. This uniformly scales all RoPE dimensions, which impairs the model's ability to understand small, local relationships.
2. NTK-aware scaling: adjusts the RoPE base frequency to stretch dimensions non-uniformly.

The method

The approach combines NTK-by-parts interpolation with an attention temperature scaling:

• NTK-by-parts: instead of scaling every RoPE dimension equally, the interpolation pressure is spread across dimensions — high frequencies are scaled less (preserving local detail), while low frequencies are scaled more (enabling longer reach).
• Attention temperature: a scaling factor 1/sqrt(t) is applied to the attention dot product to compensate for the changed distribution of attention scores, without modifying the attention code itself.

Static vs. dynamic scaling

• Static: the scaling factor s is fixed at deployment time (e.g., s=4.0 for 1M context). It is constant regardless of input length, which may impact performance on shorter texts. All major open-source frameworks (vLLM, SGLang, TokenSpeed) implement this variant.
• Dynamic: the scale factor is updated per forward pass based on actual sequence length, degrading gracefully. Not commonly used in production deployments.

References

• Peng et al. (2023), "YaRN: Efficient Context Window Extension of Large Language Models", arXiv:2309.00071
• Blog: https://amaarora.github.io/posts/2025-09-21-rope-context-extension.html

YaRN ("Yet another RoPE extensioN") is a technique for extending the context window of RoPE-based transformers. Rather than uniformly interpolating all rotary frequencies, it applies a position-aware, NTK-style partial interpolation — preserving high-frequency (short-range) positional structure while compressing low-frequency (long-range) bands into the extended range — plus an attention-temperature correction to keep perplexity stable. It typically requires only modest long-context training data, which made it the standard recipe for scaling RoPE models to long contexts.

Mistral Large 4 uses RoPE with YaRN scaling, supporting a native 1,000,000-token (1,048,576) context window per the model card — a headline feature of the release alongside its MLA backbone and indexed-attention layers.