Parameters
4.0B
Architecture
Hybrid attention transformer: 1 full-attention layer combined with 3 sliding-window attention (SWA) layers, natively supporting up to 1M-token context
Released
21.10.2026
License
Apache License 2.0
Input Modalities
Output Modalities
Context (native)
1,048,576 tokens
Context (extended)
1,048,576 tokens
About
Spark-X2.6-4B is a compact, general-purpose text language model from SparkLLM (XHToken), open-sourced under Apache 2.0 with a native context window of up to 1M tokens (1,048,576). Released October 21, 2026, it is the successor of Spark-X2.5-4B and retains exactly the same architecture as Spark-X2.5-4B, achieving its capability gains through an improved post-training recipe rather than architectural changes.
Technical Highlights
- Stronger Coding, Agentic, and Long-Context Capabilities: Substantial improvements in agentic workflows involving complex tool use, multi-step planning, and software engineering; state-of-the-art performance among similarly sized open-source models, matching or even surpassing approximately 9B-parameter models.
- Controllable Thinking Budget: Three configurable thinking levels — none, low, high — letting applications dynamically balance reasoning quality, latency, and inference cost. Thinking is enabled by default via the chat template (
enable_thinkingtoggle).
Architecture (verified from config.json)
Dense hybrid attention transformer (class Spark2_5ForCausalLM, model type spark2_5) with 36 layers in a 3:1 pattern — 27 Sliding-Window Attention (SWA) layers (window 512) interleaved with 9 full-attention layers — natively supporting a 1,048,576-token context. Attention uses 16 heads / 4 KV heads with head dimension 256 and headwise attention output gates (sigmoid gating); full-attention layers apply partial rotary embeddings (theta 5,000,000, partial rotary factor 0.25) while sliding-window layers use theta 10,000. MLPs are GELU with intermediate size 10,240; embeddings are tied with a 131,072-token vocabulary; weights ship in BF16 (~4B parameters in safetensors).
Edge Inference and Deployment
Spark-X2.6-4B delivers inference efficiency on par with or better than comparable on-device models, including Qwen3.5-4B and Gemma-E4B. Official support: Transformers 4.57.1 (trust_remote_code=True), SGLang (tool-call parser spark25, reasoning parser qwen3), vLLM, MLX (Spark-MLX-LLM, no GGUF conversion needed), llama.cpp (native spark2_5 support from b10828), Ollama (SparkLLM/Spark-X2.6-4B), LM Studio, and Unsloth Studio (native GGUF). Fine-tuning is recommended via Llama-Factory. Example sampling settings: temperature 1.0, top_p 0.95, max_new_tokens up to 131,072.
Benchmark Scores
| Benchmark | Score | Date |
|---|---|---|
|
MCP-Atlas
general_agent
|
65.01%
|
21.10.2026 |
|
BFCL-V4
general_agent
|
99.05%
|
21.10.2026 |
|
SWE-bench Pro
coding_agent
|
67.88%
|
21.10.2026 |
|
BrowseComp
general_agent
|
56.75%
|
21.10.2026 |
Hardware Requirements
Estimated VRAM per quantization at 4,096 tokens context
| Quantization | Weights | KV Cache | Total VRAM |
|---|---|---|---|
| FP16 — 16-bit | 8.0 GB | 0.6 GB | 9.0 GB |
| INT8 — 8-bit | 4.0 GB | 0.3 GB | 4.5 GB |
| INT4 — 4-bit | 2.0 GB | 0.2 GB | 2.3 GB |
| FP4 — 4-bit | 2.0 GB | 0.2 GB | 2.3 GB |
Consumer GPUs that fit this model with a single card (INT4):
Architecture
- Attention
- Hybrid Attention (16:4)
- Layers
- 36
- Hidden size
- 2560
- Context
- 1M tokens
- Parameters
- 4000M
Source: Hugging Face config.json · Spark2_5ForCausalLM · exact layer pattern · model repo
Training Pipeline
-
1
other
Improved post-training
Spark-X2.6-4B retains exactly the same architecture as Spark-X2.5-4B and achieves its capability gains through improved post-training, starting from the XHToken/Spark-X2.6-4B-Base checkpoint. The card does not break this recipe into concrete stages (no SFT/RLHF/DPO/RL breakdown is given), but reports substantial improvements in coding, agentic workflows involving complex tool use and multi-step planning, long-context, multilingual, and reasoning. The post-trained model additionally provides three configurable thinking levels (none, low, high) balancing reasoning quality, latency, and inference cost.
Linked Resources
SparkLLM on dev.to
https://dev.to/sparkllm
XHToken/Spark-plugin
https://github.com/XHToken/Spark-plugin
XHToken/Spark-MLX-LLM
https://github.com/XHToken/Spark-MLX-LLM
ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp
XHToken/LlamaFactory
https://github.com/XHToken/LlamaFactory
Ollama
https://ollama.com/download
LM Studio
https://lmstudio.ai/download
Unsloth
https://unsloth.ai/
Citation (BibTeX)
https://huggingface.co/XHToken/Spark-X2.6-4B
Spark-X2.6
https://huggingface.co/collections/XHToken/spark-x26
No comments yet
Be the first to share your thoughts!