Qwen3-Next-80B-A3B-Thinking

About

Qwen3-Next-80B-A3B-Thinking is the first reasoning model of the Qwen3-Next series, Alibaba Qwen's next-generation foundation models built for ultimate training and inference efficiency. It is a causal Mixture-of-Experts Transformer with a hybrid attention layout that replaces standard attention by interleaving Gated DeltaNet (a gated linear-attention component) with Gated Attention in a 12 x (3 DeltaNet : 1 Gated Attention) pattern, enabling efficient context modeling for ultra-long context. The high-sparsity MoE activates only 3B of its 80B total parameters per token (512 experts, 10 routed + 1 shared, expert intermediate dimension 512, hidden dimension 2048, 48 layers), drastically reducing FLOPs per token while preserving model capacity.

The model was pretrained on 15T tokens and post-trained with GSPO (Group Sequence Policy Optimization), the RL algorithm that made the hybrid attention + high-sparsity MoE combination stable to train. Multi-Token Prediction (MTP) boosts pretraining performance and accelerates inference via speculative decoding. Qwen3-Next-80B-A3B-Thinking supports a native context of 262,144 tokens, extensible up to 1,010,000 tokens with YaRN RoPE scaling (factor 4.0). It runs in thinking mode only, targeting highly complex reasoning tasks, and serves English and Chinese plus other languages (multilingual benchmarks: MultiIF, MMLU-ProX, INCLUDE, PolyMATH). It surpasses Qwen3-30B-A3B-Thinking-2507, Qwen3-32B-Thinking and the proprietary Gemini-2.5-Flash-Thinking across multiple benchmarks. Maintained by the Qwen team (Alibaba); weights on HuggingFace and Ollama under an Apache-2.0 license.

Training Data Pre-training on 15T tokens & post-training (GSPO-based RL).