Kolibri-1

Architecture

Training Pipeline

1. 1
   pretraining

   Pre-training

   Trained from random initialization with a causal next-token-prediction objective on a filtered, bilingual (German/English) corpus of 20T tokens (~62.5% English, ~23.9% German, ~13.6% code) combining curated web data, synthetic rephrasings and translations, and high-quality curated sources. Training examples were 16,384-token sequences with multiple documents packed per sequence, run over 264,750 optimization steps with two gradient-accumulation steps across 768 NVIDIA B200 GPUs (EP8 FSDP16 DP6, TorchTitan). Data curation included URL and heuristic filtering, exact/MinHash/substring deduplication, quality classification into token-mass quantile buckets, PII redaction, and mix selection via mix-search proxy models.

   511 h

2. 2
   cpt

   Mid-training

   Continued pre-training on 3.44T tokens with the sequence length increased to 65,536 tokens, using the same setup as pre-training. The data mix shifted towards instruction/reasoning, STEM QA-style, code, and agentic code/tool-use data, was globally shuffled with per-dataset and cross-dataset exact deduplication, and was decontaminated against the evaluation suite. The phase resumed the pre-training optimizer state and therefore used no additional learning-rate warmup.

   120 h

3. 3
   cpt

   Long-context extension

   Final long-context adaptation stage on 201B tokens with the sequence length increased to 262,144 tokens, the model's native context length. The mix was built by length-bucketing the mid-training pool and blending it with OCR'd PDFs (fixed at 34% of tokens, with documents up to 256k tokens; ablations found the optimal blend at x=33), and was decontaminated against the evaluation suite. It again used no additional warmup and ran with FSDP128 DP6 parallelism instead of the pre-training setup.

   13 h

4. 4
   sft

   Supervised Fine-Tuning (SFT)

   Fine-tuned the Kolibri Base checkpoint for reasoning and instruction following on a filtered, bilingual mix of permissively available open-source datasets, in-house generated data, and curated RL warm-start sets (e.g., 11k high-reward retrieval completions and filtered Merlin-Arthur rollouts), unified, decontaminated (0.006% of rows removed) and filtered for political bias. Mix weighting was chosen by adapting MergeMix: one specialist per 20 data cluster was trained on 4.5B packed positions, 69 Dirichlet-sampled weightings were merged and scored on 16 benchmarks across seven capabilities, and the best mixtures were validated at proxy scale before the target run, which saw the weighted mix about once over 168B tokens. The optimizer split of pre-training was kept (Muon for 2D backbone, AdamW elsewhere) at half the pre-training learning rates, and the final checkpoint was produced by model souping two checkpoints trained for the same token horizon on two different data mixtures.

5. 5
   rl

   Reinforcement Learning (RL)

   Trained in an asynchronous RL setup where vLLM inference ran in parallel to training with weight synchronization not waiting for ongoing generations, keeping the SFT optimizer split but with an adapted learning rate, and additionally using quantization-aware training to enable efficient low-precision (FP8) vLLM inference. Environments covered reasoning, agentic, software-engineering/terminal, tool-calling, retrieval-augmented QA, long-context QA, German-focused and Merlin-Arthur abstention tasks, with verifiable answer rewards plus format rewards, heavy randomization over harnesses, system prompts and languages, and on-policy self-distillation on hard tasks. Reasoning-effort control was trained across all levels with different length penalties, and a German language-consistency reward was applied to reasoning traces and answers.

Training & Evaluation Datasets

Linked Resources