Hy4 preview

About

Hy4 preview (tencent/Hy4-preview, released August 28, 2026 under the Apache 2.0 License) is a new-generation Mixture-of-Experts flagship model from the Tencent Hy Team: 770B total parameters, 49B activated per token, 78 backbone layers (first layer dense FFN, remaining 77 MoE with 256 routed experts and 1 shared expert each; every token activates top-8 routed experts plus the shared expert), plus 1 native MTP layer (10B total / 0.7B activated) for speculative decoding.

Its attention module, inspired by DeepSeek and GLM, employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse (64 attention heads, query compression 2048, KV compression 512, indexer 32 heads/128 dim with top-k 2048); the residual pathway uses iHC (identity Hyper-Connections) expanding inter-layer information flow across 4 residual streams. Hy4 preview scaled on three fronts vs Hy3 - model size, 1M-token context, and substantially larger pre-training and post-training data - the largest generation-over-generation gain the team has measured, putting it at the open-source frontier. Training data was built with Tencent experts (software engineering, game development, finance, security) and co-designed with CodeBuddy and WorkBuddy; in blind side-by-side evaluation 163 internal experts rated it slightly ahead of GLM 5.3 and Kimi K3 on 203 engineering tasks. Reasoning mode defaults to high (deep CoT); recommended sampling temperature 0.9, top_p 1.0.

Training Data Scaled on three fronts vs Hy3: model size, 1M-token context length, and substantially larger pre-training + post-training data. Co-designed with Tencent products (CodeBuddy, WorkBuddy) and internal domain experts (software engineering, game development, finance analysis, security). Reasoning mode defaults to high (deep CoT); recommended sampling temperature=0.9, top_p=1.0.