Parameters
2.0B
Architecture
Hybrid attention transformer: 1 full-attention layer combined with 3 sliding-window attention (SWA) layers, natively supporting up to 1M-token context
Released
01.09.2026
License
Apache License 2.0
Input Modalities
Output Modalities
Context (native)
1,048,576 tokens
Context (extended)
1,048,576 tokens
About
Spark-X2.5-1.7B is iFLYTEK's compact, general-purpose on-device language model (released by the XHToken team), designed to make capable AI practical, efficient and accessible across conversation, writing, translation, reasoning, coding, tool use and agentic workflows. It achieves leading results among open-source models of comparable size and was trained on Huawei Ascend clusters.
Its efficiency-oriented hybrid attention architecture combines one full-attention layer with three Sliding-Window Attention (SWA) layers (21 sliding + 7 full of 28 layers, sliding window 512), with Gated Attention output gates (headwise sigmoid gating, 8 Q / 2 KV heads, head dim 256, RoPE theta 5M with partial rotary 0.25 on full layers). This hybrid SWA-full attention (3:1) design substantially reduces the computational overhead of long-context models while natively supporting a context window of up to 1M tokens (1,048,576 positions). The model has ~1.7B parameters (hidden 2048, FFN 6656, vocab 131,072, tied embeddings) and supports more than 200 languages.
Pre-training used ~20 trillion tokens of web pages, books, academic publications, code and encyclopedic materials, with a dedicated long-context stage of hundreds of billions of tokens at sequence lengths up to 1M. Post-training: supervised fine-tuning, then large-scale reinforcement learning across capability domains (language understanding, reasoning, programming, tool-augmented agentic behavior, instruction following), and consolidation of the resulting domain-specialized teacher policies into one deployable model via Multi-Teacher On-Policy Distillation (MOPD).
It integrates with popular agent harnesses (Codex, Claude Code, OpenClaw, Hermes), runs on NVIDIA, Huawei, Hygon and HOUMO.AI hardware via vLLM, SGLang, llama.cpp, MLX, Ollama and LM Studio, and can be fine-tuned with LLaMA-Factory. Released September 1, 2026 under Apache 2.0.
Training Data Pretrained on ~20 trillion tokens (web pages, books, academic publications, code, encyclopedic materials) with dedicated long-context training up to 1M sequence length; post-training: supervised fine-tuning, large-scale reinforcement learning across capability domains, and MOPD consolidation of domain-specialized teacher policies. Trained on Huawei Ascend clusters. Supports 200+ languages.
Benchmark Scores
| Benchmark | Score | Date | Standing |
|---|---|---|---|
|
BFCL-V4
general_agent
|
39.57%
|
— |
#22/27
better than 19%
|
|
TAU2-Bench
general_agent
|
60.74%
|
— |
#13/21
better than 40%
|
|
TAU3-Bench
general_agent
|
27.92%
|
— |
#11/21
better than 50%
|
|
Workspace Bench
|
47.66%
|
— |
#4/5
better than 25%
|
|
VITA-Bench
general_agent
|
10.10%
|
— |
#13/16
better than 20%
|
|
BrowseComp
general_agent
|
29.89%
|
— |
#34/45
better than 25%
|
|
SWE-bench Pro
coding_agent
|
11.59%
|
— |
#71/77
better than 8%
|
|
SWE-bench Verified
coding_agent
|
32.00%
|
— |
#53/60
better than 12%
|
|
SWE-bench Multilingual
coding_agent
|
21.66%
|
— |
#45/48
better than 6%
|
|
SciCode
|
23.24%
|
— |
#23/27
better than 15%
|
|
Gaokao 2026
|
61.45%
|
— |
#5/8
better than 43%
|
|
AIME 26
stem_reasoning
|
61.99%
|
— |
#38/45
better than 16%
|
|
HMMT Feb 26
stem_reasoning
|
36.14%
|
— |
#39/44
better than 12%
|
|
IMOAnswerBench
stem_reasoning
|
32.39%
|
— |
#28/30
better than 7%
|
|
IFBench
instruction_following
|
70.89%
|
— |
#29/42
better than 29%
|
|
Humanity's Last Exam
stem_reasoning
|
7.95%
|
— |
#66/71
better than 7%
|
|
GPQA
reasoning
|
46.30%
|
— |
#10/14
better than 23%
|
|
MCPMark
general_agent
|
2.30
|
— |
#15/15
better than 0%
|
|
MCP-Atlas
general_agent
|
14.10%
|
— |
#51/54
better than 6%
|
|
IFEval
instruction_following
|
88.30%
|
— |
#16/26
better than 40%
|
|
AA-LCR
long_context
|
28.93%
|
— |
#37/42
better than 10%
|
What is a good BFCL-V4 score? What is a good TAU2-Bench score? What is a good TAU3-Bench score?
Hardware Requirements
Estimated VRAM per quantization at 4,096 tokens context
| Quantization | Weights | KV Cache | Total VRAM | On your hardware |
|---|---|---|---|---|
| FP16 — 16-bit | 4.0 GB | 0.3 GB | 4.5 GB | Which GPUs can run it? Calculate on your GPU |
| INT8 — 8-bit | 2.0 GB | 0.2 GB | 2.3 GB | Which GPUs can run it? Calculate on your GPU |
| INT4 — 4-bit | 1.0 GB | 0.1 GB | 1.1 GB | Which GPUs can run it? Calculate on your GPU |
| FP4 — 4-bit | 1.0 GB | 0.1 GB | 1.1 GB | Which GPUs can run it? Calculate on your GPU |
Consumer GPUs that fit this model with a single card (INT4):
Architecture
- Attention
- Hybrid Attention (8:2)
- Layers
- 28
- Hidden size
- 2048
- Context
- 1M tokens
- Parameters
- 2000M
Source: Hugging Face config.json · Spark2_5ForCausalLM · exact layer pattern · model repo
Training Pipeline
-
1
pretraining
Pre-training on ~20T tokens
Diverse corpus (web pages, books, academic publications, code, encyclopedic materials) with data-mixture studies balancing mathematics, logic, code and other high-value domains.
-
2
cpt
Dedicated long-context training to 1M
Hundreds of billions of tokens with sequence lengths extending to 1M tokens.
-
3
sft
Supervised fine-tuning
SFT on a carefully curated corpus; establishes instruction following, structured generation and task completion; stable policy initialization for RL.
-
4
rl
Large-scale RL across capability domains
RL across language understanding, reasoning, programming, tool-augmented agentic behavior and instruction following, yielding domain-specialized teacher policies.
-
5
other
MOPD consolidation
Complementary strengths of the domain-specialized teacher policies consolidated into a single deployable model via Multi-Teacher On-Policy Distillation.
Training & Evaluation Datasets
| Name | Role | Size | Modalities | Collection |
|---|---|---|---|---|
| Web pages | pretraining | — | — | |
| Books | pretraining | — | — | |
| Academic publications | pretraining | — | — | |
| Code | pretraining | — | — | |
| Encyclopedic materials | pretraining | — | — |
Linked Resources
XHToken/llama.cpp (Spark-X2.5 fork for llama.cpp/Ollama builds)
https://github.com/XHToken/llama.cpp
Spark-X2.5 (HuggingFace collection)
https://huggingface.co/collections/XHToken/spark-x25
XHToken on Hugging Face (all Spark models)
https://huggingface.co/XHToken
BibTeX citation (SparkLLM Team, 2026)
https://huggingface.co/XHToken/Spark-X2.5-1.7B
Trend Analysis
24h Change
+0.4%
7d Change
+2.4%
Current
1,411
downloads
-1.4%
downloads_all_time
+0.3%
likes
+0.0%
No comments yet
Be the first to share your thoughts!