Parameters
4.0B
Architecture
Hybrid attention transformer: 1 full-attention layer combined with 3 sliding-window attention (SWA) layers, natively supporting up to 1M-token context
Released
01.09.2026
License
Apache License 2.0
Input Modalities
Output Modalities
Context (native)
1,048,576 tokens
Context (extended)
1,048,576 tokens
About
Spark-X2.5-4B is a compact, general-purpose text language model from SparkLLM (XHToken), open-sourced under Apache 2.0 with a native context window of up to 1M tokens.
Technical Highlights
- Efficient Architecture and Native 1M-token Context: A hybrid attention architecture combining one full-attention layer with three sliding-window attention layers. This design substantially reduces the computational overhead typically associated with long-context models while natively supporting a context window of up to 1M tokens.
- Strong Coding and Agent Capabilities: Deeply integrated with popular agent harnesses including Codex, Claude Code, OpenClaw, and Hermes. State-of-the-art performance among models of comparable size across everyday coding, agentic workflows, reasoning, and instruction-following tasks.
- Broad Hardware and Software Compatibility: Supports NVIDIA, Huawei, Hygon, HOUMO.AI, and other hardware platforms. Compatible with vLLM, SGLang, llama.cpp, and MLX; deployable via Ollama and LM Studio. Customizable with LLaMA-Factory. Delivers superior TTFT, TOPT, and overall inference efficiency compared with similarly sized models.
- Advanced Training Algorithms: Trained on Huawei Ascend clusters. Large-scale reinforcement learning and post-training techniques such as MOPD significantly enhance reasoning, coding, agentic, and instruction-following capabilities.
Trained on ~20T tokens with large-scale reinforcement learning and MOPD consolidation, it supports 200+ languages and posts leading results among similarly sized open models on agentic, coding, math, and instruction-following benchmarks. Aimed at agentic workflows, coding, reasoning, and long-context tasks, it is built for practical, efficient on-device deployment.
Training Data Pretrained on ~20 trillion tokens (web pages, books, academic publications, code, encyclopedic materials) with dedicated long-context training up to 1M sequence length; post-training: supervised fine-tuning, large-scale reinforcement learning across capability domains, and MOPD consolidation of domain-specialized teacher policies. Trained on Huawei Ascend clusters. Supports 200+ languages.
Benchmark Scores
| Benchmark | Score | Date | Standing |
|---|---|---|---|
|
TAU3-Bench
general_agent
|
43.13%
|
— |
#9/21
better than 60%
|
|
VITA-Bench
general_agent
|
53.89%
|
— |
#8/16
better than 53%
|
|
SWE-bench Verified
coding_agent
|
47.25%
|
— |
#50/60
better than 17%
|
|
SWE-bench Multilingual
coding_agent
|
57.16%
|
— |
#38/48
better than 21%
|
|
AIME 26
stem_reasoning
|
89.16%
|
— |
#29/45
better than 36%
|
|
HMMT Feb 26
stem_reasoning
|
78.63%
|
— |
#27/44
better than 40%
|
|
IMOAnswerBench
stem_reasoning
|
73.30%
|
— |
#23/30
better than 24%
|
|
Humanity's Last Exam
stem_reasoning
|
19.32%
|
— |
#57/71
better than 20%
|
|
Workspace Bench
|
100.00%
|
— |
#1/5
better than 100%
|
|
SciCode
|
54.67%
|
— |
#14/27
better than 50%
|
|
Gaokao 2026
|
96.09%
|
— |
#2/8
better than 86%
|
|
BFCL-V4
general_agent
|
82.70%
|
— |
#10/27
better than 65%
|
|
TAU2-Bench
general_agent
|
72.84%
|
— |
#9/21
better than 60%
|
|
BrowseComp
general_agent
|
42.47%
|
— |
#31/45
better than 32%
|
|
IFBench
instruction_following
|
85.04%
|
— |
#19/42
better than 56%
|
|
MCPMark
general_agent
|
12.91%
|
— |
#11/15
better than 21%
|
|
GPQA
reasoning
|
84.24%
|
— |
#5/14
better than 69%
|
|
MCP-Atlas
general_agent
|
54.83%
|
— |
#43/54
better than 21%
|
|
IFEval
instruction_following
|
93.95%
|
— |
#9/26
better than 68%
|
|
SWE-bench Pro
coding_agent
|
49.50%
|
— |
#54/77
better than 30%
|
|
AA-LCR
long_context
|
67.02%
|
— |
#32/42
better than 24%
|
What is a good TAU3-Bench score? What is a good VITA-Bench score? What is a good SWE-bench Verified score?
Hardware Requirements
Estimated VRAM per quantization at 4,096 tokens context
| Quantization | Weights | KV Cache | Total VRAM | On your hardware |
|---|---|---|---|---|
| FP16 — 16-bit | 8.0 GB | 0.6 GB | 9.0 GB | Which GPUs can run it? Calculate on your GPU |
| INT8 — 8-bit | 4.0 GB | 0.3 GB | 4.5 GB | Which GPUs can run it? Calculate on your GPU |
| INT4 — 4-bit | 2.0 GB | 0.2 GB | 2.3 GB | Which GPUs can run it? Calculate on your GPU |
| FP4 — 4-bit | 2.0 GB | 0.2 GB | 2.3 GB | Which GPUs can run it? Calculate on your GPU |
Consumer GPUs that fit this model with a single card (INT4):
Architecture
- Attention
- Hybrid Attention (16:4)
- Layers
- 36
- Hidden size
- 2560
- Context
- 1M tokens
- Parameters
- 4000M
Source: Hugging Face config.json · Spark2_5ForCausalLM · exact layer pattern · model repo
Training Pipeline
-
1
pretraining
Pretraining (~20T tokens)
Pretrained on approximately 20 trillion tokens from web pages, books, academic publications, code, and encyclopedic materials, with data-mixture studies balancing mathematics, logic, and code; long-context capability developed in a dedicated stage with sequence lengths up to 1M tokens.
-
2
sft
Supervised fine-tuning
Supervised fine-tuning on a carefully curated corpus establishing instruction following, structured generation, and task completion, providing a stable policy initialization for reinforcement learning.
-
3
rl
Large-scale reinforcement learning
Large-scale RL across capability domains including language understanding, reasoning, programming, tool-augmented agentic behavior, and instruction following, yielding domain-specialized teacher policies.
-
4
other
MOPD policy consolidation
MOPD consolidates the complementary strengths of the domain-specialized teacher policies into a single deployable model.
Training & Evaluation Datasets
| Name | Role | Size | Modalities | Collection |
|---|---|---|---|---|
| ~20T token pre-training corpus (web, books, academic, code, encyclopedic) | pretraining | — | — | |
| SFT + large-scale RL + MOPD consolidation corpora | rl | — | — |
Linked Resources
SparkLLM Slack
https://join.slack.com/t/tokenspark/shared_invite/zt-432qf8l2f-5~dLyXv8uETr0P0UuC07nw
SparkLLM Discord
https://discord.gg/kTDE2Hg8aw
SparkLLM YouTube
https://www.youtube.com/@SparkLLM
SparkLLM on dev.to
https://dev.to/sparkllm
SparkLLM on Bluesky
https://bsky.app/profile/sparkllm.bsky.social
SparkLLM on X (Twitter)
https://x.com/sparkllm
SparkLLM on Zhihu
https://www.zhihu.com/people/zhiikz7qh7m
Spark-plugin (vLLM/Ascend plugin)
https://github.com/XHToken/Spark-plugin.git
Spark-MLX-LLM (MLX runtime)
https://github.com/XHToken/Spark-MLX-LLM.git
llama.cpp (Spark fork for GGUF/Ollama/LM Studio)
https://github.com/XHToken/llama.cpp.git
LlamaFactory (Spark fine-tuning fork)
https://github.com/XHToken/LlamaFactory
Spark-X2.5 Hugging Face Collection
https://huggingface.co/collections/XHToken/spark-x25
Trend Analysis
24h Change
-16.7%
7d Change
-70.3%
Current
8,783
followers
+0.4%
downloads_all_time
+0.3%
likes
+0.0%
No comments yet
Be the first to share your thoughts!