ERNIE-5.5-Flash-Base

About

ERNIE 5.5 Flash is an efficient sparse Mixture-of-Experts (MoE) language model with 280B total parameters and 14B activated per token, pre-trained from scratch on 33.5T tokens with a training context of up to 128K; fine-tuning and inference support a 1M-token context. Releasing the weights at the end of pre-training lets developers build on the model's general capabilities before any mid- or post-training — ERNIE-5.5-Flash-Base can serve as a common starting point for domain adaptation, instruction tuning and reinforcement-learning research.

Efficiency was a key design goal, revisiting every component: hybrid attention (37 Heavily Compressed Attention (HCA) layers caching a single KV entry per 128 tokens, plus 6 Multi-head Latent Attention layers with sparse attention (MLA + DSA)), Multi-Router Latent MoE (512 routed experts with top-10 routing plus 1 shared expert in a 2048-d latent space, multiple independently initialized sigmoid routers), ECHO (the backbone's own final layer replaces a separate MTP module and doubles as a built-in draft model for speculative decoding), Manifold-constrained Hyper-Connections (mHC) in place of the standard residual stream, and a MultiMax output head. Pre-training used the Muon optimizer (communication-efficient distributed variant) with FlashMask-4 attention kernels; the tokenizer is a ~200K-entry BPE (numbers tokenized right-to-left in groups of up to 3 digits).

Under the paper's unified evaluation pipeline the base checkpoint delivers strong results across knowledge, reasoning, math, code, multilingual and long-context benchmarks. Available in FP8 (safetensors in this repo) and BF16 (ERNIE-5.5-Flash-Base-BF16); text-only base model served via SGLang and vLLM.