Hy4 flash
MoE multimodal decoder (MLA hybrid attention: IndexPool sparse + SWA, LatentMoE, Engram n-gram embeddings)
Parameters
220.0B / 12.0B active
MoE total / active
Context
1,048,576 tokens
native · extended 1,048,576
Modalities
text, image text
input → output
Benchmarks
#77/138
better than 44% of all scored models
Size class
#5/12
better than 58% of similar-size models
24h trend
+92.4%
Interest delta, 24h
License
Tencent Hy License
About
Tencent Hy4 flash
Hy4 flash is the Tencent Hy Team's (Hunyuan) most cost-effective Hy model to date — a 220B-parameter multimodal Mixture-of-Experts model that natively accepts text and image inputs, generates text, and supports a context window of up to 1M tokens (1,048,576). Of the 220B total parameters, 155.5B belong to the language backbone, 63.5B to the N-gram embedding (Engram) tables, 0.43B to the vision encoder (ViT), and 0.85B to the built-in DFly2 speculative-decoding drafter; only 12B parameters are active per token (11.3B text + 0.43B vision).
The 42-layer language backbone consists of one dense feed-forward layer and 41 MoE layers, with identity Hyper-Connections (iHC) over 4 residual streams supporting inter-layer information flow. A vision encoder adapted from SigLIP2-So400m-Patch16-Res512 is connected to the backbone through an MLP projector.
Compared with its large sibling Hy4, Hy4 flash introduces three notable architectural changes:
- Hybrid attention — Sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio (backbone layout: 2 × Sparse → 10 × (3 × SWA → Sparse)), formulated as Multi-head Latent Attention (MLA) with 12 sparse layers (32 attention heads) and 30 SWA layers (64 heads). Sparse context selection covers up to 2,048 tokens via group-level selection; the SWA window is 512 tokens; interleaved MRoPE encodes multimodal positions.
- LatentMoE — routed experts operate in a lower-dimensional latent space (MoE latent dimension 1,536; expert intermediate size 1,536): each token activates 16 of 512 routed experts alongside one shared expert.
- N-gram embedding (Engram) — the backbone is augmented with learned bigram and trigram representations retrieved via multi-head hash lookups (8 hash heads per N-gram order, 10.3M hash vocabulary per head, 768-dimensional embeddings per order). The embedding tables hold 63.4B parameters and are applied at backbone layers 1, 11, 21, and 31.
The hybrid attention design shrinks the KV cache to just 1/24 of Hy3 and 1/6 of Hy4. Post-training couples self-distillation from Hy4 — establishing strong general intelligence — with reinforcement learning that cuts average end-to-end output tokens by 50%. At inference, the combination of megakernels, 4-bit quantization, and DFly2 — a five-layer, 0.85B-parameter drafter that reuses the target model's Top-k Sparse KV cache and indexer selections (up to 2,048 positions) instead of maintaining its own KV cache — pushes decoding speed past 150 tokens/s at a low, constant cost, and is prefill-to-decode disaggregation ready.
On Tencent's internal WorkIndex benchmark, Hy4 flash scores 42.1 (vs 39.7 for Hy4 preview and 51.4 for Hy4) at an average cost per task of only 1/13 of Hy4 preview, 1/20 of Hy4, and 1/8 of DeepSeek-V4.1-Flash-1017 (Max tier, peak-hour pricing). In an 85-user, 259-task blind evaluation across 27 categories, user satisfaction was 2.79/4 — close to Claude Haiku 5.5's 2.81/4 — with the lowest cost per task (RMB 0.26) and the shortest median end-to-end completion time (6.7 minutes) among the evaluated models. Tencent recommends Hy4 flash for everyday productivity work (deep search, document processing, slide creation, simple frontend demos, computer use) while pointing to Hy4 for complex, long-horizon software engineering.
The release ships as three open-weight artifacts on Hugging Face, ModelScope, and GitCode: Hy4 flash (this instruct model), Hy4 flash-FP8 (FP8-quantized instruct), and Hy4 flash-base (pre-trained base), plus an AngelSlim GGUF build for local testing. The model is bilingual (English and Chinese documentation, languages en/zh), supports a reasoning mode switch (reasoning_effort: "high" thinking default, no_think for direct responses) with recommended sampling parameters temperature=0.9 and top_p=1.0, tool calling, vLLM and SGLang serving, a full finetuning pipeline, and AngelSlim-based quantization. Tencent launched Hy4 flash through its apps and cloud API on 2026-11-16, publishing the open weights on 2026-11-18. It is released under the Tencent Hy License.
Training Data Base model was exposed to extensive synthetic instruction data during mid-training (card disclaimer); post-training used self-distillation from Hy4 plus reinforcement learning, cutting average end-to-end output tokens by 50%.
Benchmark Scores
| Benchmark | Score | Date | Standing |
|---|---|---|---|
|
MMLU-Redux
knowledge
|
66.39%
|
— |
#25/35
better than 29%
|
|
INCLUDE
multilingual
|
79.73%
|
— |
#7/18
better than 59%
|
|
C³
reasoning
|
76.19%
|
— |
#2/3
better than 50%
|
|
PIQA
reasoning
|
95.50%
|
— |
#2/7
better than 83%
|
|
OCW Courses
stem_reasoning
|
14.29%
|
— |
#2/3
better than 50%
|
|
CMMLU
knowledge
|
100.00%
|
— |
#1/4
better than 100%
|
|
TextVQA
multimodal
|
50.00%
|
— |
#2/5
better than 75%
|
|
Chinese-SimpleQA
knowledge
|
57.45%
|
— |
#2/3
better than 50%
|
|
SimpleVQA
vision_language
|
32.71%
|
— |
#16/18
better than 12%
|
|
CountBenchQA
multimodal
|
91.49%
|
— |
#2/5
better than 75%
|
|
AGIEval (English)
reasoning
|
75.40
|
— |
#3/3
better than 0%
|
|
MMMU-Pro
vision_language
|
44.37%
|
— |
#34/39
better than 13%
|
|
U-MATH
stem_reasoning
|
100.00%
|
— |
#1/3
better than 100%
|
|
C-Eval
knowledge
|
87.50%
|
— |
#7/19
better than 67%
|
|
HellaSwag
reasoning
|
63.09%
|
— |
#6/10
better than 44%
|
|
ARC-c
reasoning
|
94.49%
|
— |
#5/9
better than 50%
|
|
MMLU
knowledge
|
86.35%
|
— |
#12/19
better than 33%
|
|
WorkIndex (Tencent internal)
general_agent
|
100.00%
|
— |
#1/2
better than 100%
|
|
RealWorldQA
vision_language
|
59.53%
|
— |
#19/24
better than 22%
|
|
SimpleQA
knowledge
|
29.81%
|
— |
#2/3
better than 50%
|
|
MMLU-Pro
knowledge
|
35.81%
|
— |
#38/50
better than 24%
|
|
SocialIQA
reasoning
|
100.00%
|
— |
#1/7
better than 100%
|
|
WorkBuddy Blind User Evaluation (satisfaction)
general_capabilities
|
100.00%
|
— |
#1/1
only tracked model
|
|
CV-Bench
vision_language
|
83.40
|
— |
#3/3
better than 0%
|
|
AGIEval (Chinese)
reasoning
|
100.00%
|
— |
#1/3
better than 100%
|
|
SuperGPQA
knowledge
|
71.17%
|
— |
#13/30
better than 55%
|
|
MBPP+
code
|
73.70
|
— |
#5/5
better than 0%
|
|
KOR-Bench
reasoning
|
4.87%
|
— |
#6/7
better than 17%
|
|
Belebele (5-language subset)
multilingual
|
41.67%
|
— |
#5/7
better than 17%
|
What is a good MMLU-Redux score? What is a good INCLUDE score? What is a good C³ score?
Hardware Requirements
Estimated VRAM per quantization at 4,096 tokens context — MoE: all experts must fit in VRAM, only 12.0B are active per token
| Quantization | Weights | KV Cache | Total VRAM | On your hardware |
|---|---|---|---|---|
| FP16 — 16-bit | 440.0 GB | 1.9 GB | 463.9 GB | Which GPUs can run it? Calculate on your GPU |
| INT8 — 8-bit | 220.0 GB | 0.9 GB | 231.9 GB | Which GPUs can run it? Calculate on your GPU |
| INT4 — 4-bit | 110.0 GB | 0.5 GB | 116.0 GB | Which GPUs can run it? Calculate on your GPU |
| FP4 — 4-bit | 110.0 GB | 0.5 GB | 116.0 GB | Which GPUs can run it? Calculate on your GPU |
Only datacenter GPUs — FP16 needs 463.9 GB. For consumer hardware, choose the INT4 build with a multi-GPU setup.
Architecture
multimodal MoE decoder
- Parameters
- 220B
- Active params
- 12B
- Layers
- 42 12× Full Attention + 30× Sliding Window Attn
- Attention
- MLA Sparse Attention
- Experts
- 512 top-16 + 1 shared
- Hidden
- 3072
- Head dim
- 64
- Vocabulary
- 121K
- Context
- 1M
Source: Hugging Face config.json · HYV4FlashForConditionalGeneration · exact layer pattern · model repo
Training Pipeline
-
1
pretraining
Pre-training
Hy4 flash's pre-training integrates three architectural innovations: hybrid attention interleaving IndexPool sparse attention and sliding-window attention (SWA) at a 1:3 ratio (2x Sparse -> 10x (3x SWA -> Sparse) across the 42-layer backbone), LatentMoE with routed experts in a lower-dimensional latent space, and Engram N-gram embedding (learned bigram/trigram representations via multi-head hash lookups). This design reduces KV cache size to 1/24 of Hy3 and 1/6 of Hy4, with extensive scaling ladder experiments ensuring performance meets expectations. The card also notes the base model was exposed to a large amount of synthetic instruction data during mid-training, which occasionally causes it to miss the expected answer format in few-shot evaluations.
-
2
sft
Self-distillation from Hy4 (post-training)
Post-training uses self-distillation from Hy4 to establish strong general intelligence. The card documents only this one-line rationale; no data composition, teacher-student mechanics, or loss details are given. Mapped to 'sft' as the closest allowed stage type for this distillation/alignment stage.
-
3
rl
Reinforcement learning (post-training)
Post-training reinforcement learning cuts average end-to-end output tokens by 50%, contributing to the model's per-task cost efficiency. No reward design, algorithm, or RL hyperparameters are documented on the page.
-
4
other
DFly2 drafter training (AngelSpec)
DFly2 is Hy4 flash's built-in 5-layer, 0.85B speculative-decoding drafter, trained with AngelSpec and built directly on the target model's Top-k Sparse KV: each draft layer shares the KV cache (KV Share) and the compressed indexer's selection of up to 2,048 positions (Index Share) with its corresponding sparse attention layer. It trains alongside/for the target model rather than as a documented sequential recipe stage, yielding constant per-draft-token cost (~2K keys max regardless of context), zero extra KV memory, and PD-disaggregation readiness.
Linked Resources
Official Website — Introducing Hy4 flash
https://hy.tencent.ai/research/hy4-flash
Tencent-Hunyuan/Hy4-flash
https://github.com/Tencent-Hunyuan/Hy4-flash
Tencent/AngelSpec
https://github.com/Tencent/AngelSpec
tencent/AngelSlim
https://github.com/tencent/AngelSlim
Hy4 flash vLLM Recipe
https://recipes.vllm.ai/tencent/Hy4-flash
Hy4 flash SGLang Cookbook
https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy4-flash
Finetuning Guide
https://huggingface.co/tencent/Hy4-flash/blob/main/finetune/README.md
Hy4 flash
https://huggingface.co/collections/tencent/hy4-flash
ModelScope mirror
https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash
GitCode mirror
https://ai.gitcode.com/tencent_hunyuan/Hy4-flash
Tencent Hy License (LICENSE)
https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE
Trend Analysis
24h Change
+92.4%
Current
3,949
likes
+59.6%
downloads_all_time
+92.4%
followers
+1.7%
No comments yet
Be the first to share your thoughts!