Sage-S2.1-Flash

Shanghai AI Laboratory

Glm5Next hybrid sparse-MoE: 34 gated linear-attention + 11 DSA full-attention layers, 288 experts (top-8 + 1 shared), native 1M context

Open Weights Commercial Use Multimodal F32,BF16 Sage

Parameters

321.3B

Context

1,048,576 tokens

native

Modalities

text, image, video text

input → output

Benchmarks

#31/134

better than 77% of all scored models

License

MIT License

About

Sage-S2.1-Flash (card title: Sage S2.1 Flash) is a scientific multimodal model from Shanghai Artificial Intelligence Laboratory (the team behind the InternLM model family). It is designed for research tasks that require multiple rounds of analysis and verification, combining scientific data understanding with information retrieval, programming, and tool use. The model can operate specialized software to perform analyses, check results against intermediate outputs and visual feedback, and adjust its subsequent steps accordingly.

Scientific multimodality via SciPerception

Through the unified representation architecture of the companion Sage SciPerception system, the model supports more than ten types of scientific input, including molecular graphs, crystal structures (CIF), and protein sequences and 3D protein structures. Features of each input type are encoded and projected into a shared representation space, where they can be reasoned over alongside natural-language instructions. Supported task styles include prediction, classification, site localization, quantitative analysis, and structure inference.

Scientific encoders with trainable projectors (for example the projectors/sage-s2.1-flash/cif/jarvis crystal projector) map scientific data into the language model's input embedding space; the projector output is combined with the text token embeddings to form the prompt sequence. On the serving side, vLLM runs with --enable-prompt-embeds and accepts prompt_embeds on /v1/completions, using the tokenizer and text embeddings that match the base model.

Backbone architecture (config.json)

config.json declares Glm5NextForConditionalGeneration with model type glm5_next: a hybrid sparse-Mixture-of-Experts (MoE) decoder with 45 layers that mixes gated linear attention and DeepSeek Sparse Attention (DSA) full attention: 34 gated linear-attention layers (the kda_layers / linear_attn_config group - a Kimi Delta Attention-style gated linear attention with short-conv kernel size 4, gate lower bound -5, 64 heads x 128 head dim) alternate with 11 DSA full-attention layers in a repeating 3:1 pattern plus one final linear layer. The DSA layers use MLA-style latent attention components (q_lora_rank 1536, kv_lora_rank 512, qk_nope_head_dim 256, v_head_dim 256, nope-only MLA with qk_rope_head_dim 0) plus a sparse indexer (index_head_dim 128, index_topk 2048, interleaved indexer RoPE) running on every full-attention layer.

MoE routing: 288 routed experts with top-8 per-token selection and 1 shared expert, sigmoid scoring via the noaux_tc top-k method (topk_method: noaux_tc, routed_scaling_factor 2.5, router in float32, norm_topk_prob on), MoE intermediate size 2048 (dense intermediate 12288) and the first 3 layers dense (first_k_dense_replace 3). Additional structural specifics: hyper-connections (mhc: true, hc_mult 4, 20 Sinkhorn iterations) as multi-stream residual connections, one multi-token prediction (MTP) layer (num_nextn_predict_layers 1, indexer shared for MTP iteration), hidden size 4,096, vocab 154,880, tie_word_embeddings false.

The vision tower (glm5_next_vision) is a 24-layer ViT-style encoder (image size 448, patch 14, spatial merge 2, SwiGLU, projection intermediate 10,240) feeding into the 4,096-dim language-model space; both image and video token ids exist in config (image_token_id 154854, video_token_id 154855), so image and video ingestion run through the shared vision stack.

Parameters and native context

Safetensors total: 321,323,031,390 parameters ("321B params"; 321,319,393,536 BF16 + 3,637,854 F32 - the F32 slice matches the float32 router dtype); tensor types F32 and BF16. max_position_embeddings 1,048,576 gives a native 1M context window. A quantized FP8 release ships as internlm/Sage-S2.1-Flash-FP8; the model tree lists 9 community quantizations, and ModelScope hosts mirrors for both releases. Downloads last month: 337.

Evaluation

The card's evaluation covers a 15-benchmark comparison chart plus a 33-row benchmark table against five comparison models: GLM-5.3-Flash, DeepSeek-V4.1-Flash-1017, Qwen3.8-Max, GPT-5.6 Sol, and Claude Opus 5. Headline results: Sage S2.1 Flash placed first on MLE-bench Lite (90.3, ahead of DeepSeek-V4.1-Flash-1017 at 89.2**)**, OfficeQA Pro (72.2), Harvey LAB (25.0) and SpreadsheetBench 2 (37.1), and second on AutomationBench, ALE-CLI, BFCL V4, WebArena-Verified, DeepResearch Bench II, JobBench, and OSWorld-Verified. Agentic coding reaches 87.9 on Terminal-Bench 2.1 and 67.9 on DeepSWE; search and deep-research runs hit 91.2 on BrowseComp, 94.4 on DeepSearchQA, 81.5 on WideSearch, and 54.2 on DeepResearch Bench II; computer-use and web tasks reach 85.2 on OSWorld-Verified, 67.6 on WebArena-Verified, 78.0 on Vision2Web, 84.8 on OSWorld-G and 75.8 on BabyVision.

Deployment

Official deployment guides cover the BF16 and FP8 releases for general tasks (coding, office productivity, security, computer use); scientific workloads use the projector flow above with shared loading scripts and the inference guide from the companion internlm/Sage-SciPerception repository (load_projector.py, modeling_projector.py, vLLM --enable-prompt-embeds). The model is not served by any HF Inference Provider.

License and release

Released on HuggingFace on 2026-11-15. The code and model weights are released under the MIT License - open weights, fully permissive, commercial use allowed.

Benchmark Scores

Benchmark Score Date Standing
Workspace-Bench-Lite
general_agent
90.65%
—
#3/6
better than 60%
BFCL-V4
general_agent
89.88%
—
#2/33
better than 97%
MLS-Bench
coding_agent
21.21%
—
#4/6
better than 40%
PaperBench
coding_agent
88.99%
—
#6/15
better than 64%
Vision2Web
vision_language
94.23%
—
#2/9
better than 88%
BrowseComp
general_agent
97.13%
—
#4/50
better than 92%
PostTrainBench
coding_agent
89.08%
—
#6/24
better than 78%
GDPVal
general_agent
86.45%
—
#6/26
better than 80%
WorkSpaceBench
general_agent
20.83%
—
#11/15
better than 29%
WebArena-Verified
general_agent
96.46%
—
#2/11
better than 90%

Hardware Requirements

Estimated VRAM per quantization at 4,096 tokens context

Quantization Weights KV Cache Total VRAM On your hardware
FP16 — 16-bit 642.6 GB 49.9 GB 724.6 GB Which GPUs can run it? Calculate on your GPU
INT8 — 8-bit 321.3 GB 24.9 GB 362.3 GB Which GPUs can run it? Calculate on your GPU
INT4 — 4-bit 160.7 GB 12.5 GB 181.2 GB Which GPUs can run it? Calculate on your GPU
FP4 — 4-bit 160.7 GB 12.5 GB 181.2 GB Which GPUs can run it? Calculate on your GPU

Only datacenter GPUs — FP16 needs 724.6 GB. For consumer hardware, choose the INT4 build with a multi-GPU setup.

Architecture

Decoder Block ×45 input Embedding vocab 155K · d 4096 Linear / Recurrent 64 heads ×34 Full Attention 64 heads ×11 MoE FFN 288 experts · top-8 · +1 shared · dᴻ 2048 MTP Head ×1 speculative layer Final Norm LM Head vocab 155K output

hybrid linear-attention + sparse (DSA) full-attention MoE transformer decoder

MoE Hybrid Vision
Parameters
321.3B
Layers
45 34× Linear / Recurrent + 11× Full Attention
Attention
MLA Sparse Attention
Experts
288 top-8 + 1 shared
Hidden
4096
Vocabulary
155K
Context
1M
silu BF16 MTP ×1 First 3 dense sigmoid noaux_tc top-8…

Source: Hugging Face config.json · Glm5NextForConditionalGeneration · exact layer pattern · model repo

Comments (0)

No comments yet

Be the first to share your thoughts!

What now?

You have read the full card for Sage-S2.1-Flash — here is where to go next.