Hy4 flash

Tencent

MoE multimodal decoder (MLA hybrid attention: IndexPool sparse + SWA, LatentMoE, Engram n-gram embeddings)

Multimodal BF16, F32, I32 Hunyuan en zh

Parameters

220.0B / 12.0B active

MoE total / active

Context

1,048,576 tokens

native · extended 1,048,576

Modalities

text, image text

input → output

Benchmarks

#77/138

better than 44% of all scored models

Size class

#5/12

better than 58% of similar-size models

24h trend

+92.4%

Interest delta, 24h

License

Tencent Hy License

About

Tencent Hy4 flash

Hy4 flash is the Tencent Hy Team's (Hunyuan) most cost-effective Hy model to date — a 220B-parameter multimodal Mixture-of-Experts model that natively accepts text and image inputs, generates text, and supports a context window of up to 1M tokens (1,048,576). Of the 220B total parameters, 155.5B belong to the language backbone, 63.5B to the N-gram embedding (Engram) tables, 0.43B to the vision encoder (ViT), and 0.85B to the built-in DFly2 speculative-decoding drafter; only 12B parameters are active per token (11.3B text + 0.43B vision).

The 42-layer language backbone consists of one dense feed-forward layer and 41 MoE layers, with identity Hyper-Connections (iHC) over 4 residual streams supporting inter-layer information flow. A vision encoder adapted from SigLIP2-So400m-Patch16-Res512 is connected to the backbone through an MLP projector.

Compared with its large sibling Hy4, Hy4 flash introduces three notable architectural changes:

  1. Hybrid attention — Sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio (backbone layout: 2 × Sparse → 10 × (3 × SWA → Sparse)), formulated as Multi-head Latent Attention (MLA) with 12 sparse layers (32 attention heads) and 30 SWA layers (64 heads). Sparse context selection covers up to 2,048 tokens via group-level selection; the SWA window is 512 tokens; interleaved MRoPE encodes multimodal positions.
  2. LatentMoE — routed experts operate in a lower-dimensional latent space (MoE latent dimension 1,536; expert intermediate size 1,536): each token activates 16 of 512 routed experts alongside one shared expert.
  3. N-gram embedding (Engram) — the backbone is augmented with learned bigram and trigram representations retrieved via multi-head hash lookups (8 hash heads per N-gram order, 10.3M hash vocabulary per head, 768-dimensional embeddings per order). The embedding tables hold 63.4B parameters and are applied at backbone layers 1, 11, 21, and 31.

The hybrid attention design shrinks the KV cache to just 1/24 of Hy3 and 1/6 of Hy4. Post-training couples self-distillation from Hy4 — establishing strong general intelligence — with reinforcement learning that cuts average end-to-end output tokens by 50%. At inference, the combination of megakernels, 4-bit quantization, and DFly2 — a five-layer, 0.85B-parameter drafter that reuses the target model's Top-k Sparse KV cache and indexer selections (up to 2,048 positions) instead of maintaining its own KV cache — pushes decoding speed past 150 tokens/s at a low, constant cost, and is prefill-to-decode disaggregation ready.

On Tencent's internal WorkIndex benchmark, Hy4 flash scores 42.1 (vs 39.7 for Hy4 preview and 51.4 for Hy4) at an average cost per task of only 1/13 of Hy4 preview, 1/20 of Hy4, and 1/8 of DeepSeek-V4.1-Flash-1017 (Max tier, peak-hour pricing). In an 85-user, 259-task blind evaluation across 27 categories, user satisfaction was 2.79/4 — close to Claude Haiku 5.5's 2.81/4 — with the lowest cost per task (RMB 0.26) and the shortest median end-to-end completion time (6.7 minutes) among the evaluated models. Tencent recommends Hy4 flash for everyday productivity work (deep search, document processing, slide creation, simple frontend demos, computer use) while pointing to Hy4 for complex, long-horizon software engineering.

The release ships as three open-weight artifacts on Hugging Face, ModelScope, and GitCode: Hy4 flash (this instruct model), Hy4 flash-FP8 (FP8-quantized instruct), and Hy4 flash-base (pre-trained base), plus an AngelSlim GGUF build for local testing. The model is bilingual (English and Chinese documentation, languages en/zh), supports a reasoning mode switch (reasoning_effort: "high" thinking default, no_think for direct responses) with recommended sampling parameters temperature=0.9 and top_p=1.0, tool calling, vLLM and SGLang serving, a full finetuning pipeline, and AngelSlim-based quantization. Tencent launched Hy4 flash through its apps and cloud API on 2026-11-16, publishing the open weights on 2026-11-18. It is released under the Tencent Hy License.

Training Data Base model was exposed to extensive synthetic instruction data during mid-training (card disclaimer); post-training used self-distillation from Hy4 plus reinforcement learning, cutting average end-to-end output tokens by 50%.

Benchmark Scores

Benchmark Score Date Standing
MMLU-Redux
knowledge
66.39%
—
#25/35
better than 29%
INCLUDE
multilingual
79.73%
—
#7/18
better than 59%
C³
reasoning
76.19%
—
#2/3
better than 50%
PIQA
reasoning
95.50%
—
#2/7
better than 83%
OCW Courses
stem_reasoning
14.29%
—
#2/3
better than 50%
CMMLU
knowledge
100.00%
—
#1/4
better than 100%
TextVQA
multimodal
50.00%
—
#2/5
better than 75%
Chinese-SimpleQA
knowledge
57.45%
—
#2/3
better than 50%
SimpleVQA
vision_language
32.71%
—
#16/18
better than 12%
CountBenchQA
multimodal
91.49%
—
#2/5
better than 75%

Hardware Requirements

Estimated VRAM per quantization at 4,096 tokens context — MoE: all experts must fit in VRAM, only 12.0B are active per token

Quantization Weights KV Cache Total VRAM On your hardware
FP16 — 16-bit 440.0 GB 1.9 GB 463.9 GB Which GPUs can run it? Calculate on your GPU
INT8 — 8-bit 220.0 GB 0.9 GB 231.9 GB Which GPUs can run it? Calculate on your GPU
INT4 — 4-bit 110.0 GB 0.5 GB 116.0 GB Which GPUs can run it? Calculate on your GPU
FP4 — 4-bit 110.0 GB 0.5 GB 116.0 GB Which GPUs can run it? Calculate on your GPU

Only datacenter GPUs — FP16 needs 463.9 GB. For consumer hardware, choose the INT4 build with a multi-GPU setup.

Architecture

Decoder Block ×42 input Embedding vocab 121K · d 3072 Full Attention 32 heads · dₕ 64 · win 512 ×12 Sliding Window Attn 32 heads · dₕ 64 · win 512 ×30 MoE FFN 512 experts · top-16 · +1 shared · dᴻ 1536 Final Norm LM Head vocab 121K output

multimodal MoE decoder

MoE Vision
Parameters
220B
Active params
12B
Layers
42 12× Full Attention + 30× Sliding Window Attn
Attention
MLA Sparse Attention
Experts
512 top-16 + 1 shared
Hidden
3072
Head dim
64
Vocabulary
121K
Context
1M
silu BF16 SWA 512 First 1 dense sigmoid top-k routing…

Source: Hugging Face config.json · HYV4FlashForConditionalGeneration · exact layer pattern · model repo

Training Pipeline

  1. 1
    pretraining

    Pre-training

    Hy4 flash's pre-training integrates three architectural innovations: hybrid attention interleaving IndexPool sparse attention and sliding-window attention (SWA) at a 1:3 ratio (2x Sparse -> 10x (3x SWA -> Sparse) across the 42-layer backbone), LatentMoE with routed experts in a lower-dimensional latent space, and Engram N-gram embedding (learned bigram/trigram representations via multi-head hash lookups). This design reduces KV cache size to 1/24 of Hy3 and 1/6 of Hy4, with extensive scaling ladder experiments ensuring performance meets expectations. The card also notes the base model was exposed to a large amount of synthetic instruction data during mid-training, which occasionally causes it to miss the expected answer format in few-shot evaluations.

  2. 2
    sft

    Self-distillation from Hy4 (post-training)

    Post-training uses self-distillation from Hy4 to establish strong general intelligence. The card documents only this one-line rationale; no data composition, teacher-student mechanics, or loss details are given. Mapped to 'sft' as the closest allowed stage type for this distillation/alignment stage.

  3. 3
    rl

    Reinforcement learning (post-training)

    Post-training reinforcement learning cuts average end-to-end output tokens by 50%, contributing to the model's per-task cost efficiency. No reward design, algorithm, or RL hyperparameters are documented on the page.

  4. 4
    other

    DFly2 drafter training (AngelSpec)

    DFly2 is Hy4 flash's built-in 5-layer, 0.85B speculative-decoding drafter, trained with AngelSpec and built directly on the target model's Top-k Sparse KV: each draft layer shares the KV cache (KV Share) and the compressed indexer's selection of up to 2,048 positions (Index Share) with its corresponding sparse attention layer. It trains alongside/for the target model rather than as a documented sequential recipe stage, yielding constant per-draft-token cost (~2K keys max regardless of context), zero extra KV memory, and PD-disaggregation readiness.

Comments (0)

No comments yet

Be the first to share your thoughts!

What now?

You have read the full card for Hy4 flash — here is where to go next.