Qwen3-235B-A22B

Qwen Team (Alibaba Cloud)

Parameters

235.0B total / 22.0B active

MoE: total / active

Architecture

Mixture-of-Experts Transformer with Thinking mode

Released

27.04.2025

License

Apache License 2.0

Open Weights Commercial Use Multimodal bfloat16 Qwen3 en zh multi

Input Modalities

text

Output Modalities

text

Context (native)

32,768 tokens

Context (extended)

131,072 tokens

Openness Index Score 100.0/100

About

Qwen3-235B-A22B is the flagship sparse Mixture-of-Experts model of Alibaba's Qwen3 generation - a causal decoder-only LLM with 235B total parameters (234B non-embedding), 22B activated per token, and 94 layers. Each MoE layer routes each token to 8 of 128 experts (expert dim 1536; dense FFN 12288 on non-MoE layers), with Grouped-Query Attention (GQA) of 64 query heads and 4 KV heads (head dim 128), hidden size 4096, 152K vocabulary and RoPE theta 1M.

Its signature capability is seamless thinking-mode switching: one model serves both a thinking mode (for complex logical reasoning, math, and coding) and a non-thinking mode (for efficient general-purpose dialogue), set via the chat template (enable_thinking) or no_think in the prompt. Qwen3 delivers significantly enhanced reasoning (surpassing QwQ and Qwen2.5 in both modes), superior human-preference alignment for creative writing, role-play and multi-turn dialogue, and expertise in agentic tool integration in both modes - leading open-source performance in complex agent-based tasks. It supports 100+ languages and dialects with strong multilingual instruction following and translation.

Pre-trained on ~36 trillion tokens across 119 languages with a four-stage post-training pipeline, it has a native context of 32,768 tokens extendable to 131,072 with YaRN. Released April 27, 2025 under Apache 2.0.

Training Data Pretrained on ~36 trillion tokens across 119 languages; 4-stage post-training pipeline

Benchmark Scores

Benchmark Score Date Standing
MMMU-Pro
vision_language
74.35%
01.02.2026
#24/35
better than 29%
MMLU-Pro
knowledge
78.71%
01.02.2026
#20/48
better than 60%
MMLU-Redux
knowledge
92.44%
01.02.2026
#5/30
better than 86%
SuperGPQA
knowledge
87.16%
01.02.2026
#7/28
better than 78%
Multi-Challenge
instruction_following
35.65%
01.02.2026
#16/18
better than 12%
LongBench v2
long_context
67.65%
01.02.2026
#9/22
better than 62%
Humanity's Last Exam
stem_reasoning
30.49%
01.02.2026
#53/71
better than 26%
GPQA Diamond
stem_reasoning
75.04%
01.02.2026
#60/80
better than 25%
HMMT Feb 25
stem_reasoning
78.44%
01.02.2026
#11/17
better than 38%
HMMT Nov 25
stem_reasoning
64.47%
01.02.2026
#13/20
better than 37%

Showing 10 of 76 results

What is a good MMMU-Pro score? What is a good MMLU-Pro score? What is a good MMLU-Redux score?

Hardware Requirements

Estimated VRAM per quantization at 4,096 tokens context — MoE: all experts must fit in VRAM, only 22.0B are active per token

Quantization Weights KV Cache Total VRAM On your hardware
FP16 — 16-bit 470.0 GB 3.4 GB 496.9 GB Which GPUs can run it? Calculate on your GPU
INT8 — 8-bit 235.0 GB 1.7 GB 248.5 GB Which GPUs can run it? Calculate on your GPU
INT4 — 4-bit 117.5 GB 0.9 GB 124.2 GB Which GPUs can run it? Calculate on your GPU
FP4 — 4-bit 117.5 GB 0.9 GB 124.2 GB Which GPUs can run it? Calculate on your GPU

Only datacenter GPUs — FP16 needs 496.9 GB. For consumer hardware, choose the INT4 build with a multi-GPU setup.

Architecture

Decoder Block input Embedding vocab 152K · d 4096 Full Attention GQA 64:4 · dₕ 128 ×94 MoE FFN 128 experts · top-8 · dᴻ 1536 Final Norm LM Head vocab 152K output
Attention
Grouped Query Attention (64:4)
MoE
128 experts · top-8 per token
Layers
94
Hidden size
4096
Context
41K tokens
RoPE θ
1M
Parameters
235000M
Active params
22000M

Source: Hugging Face config.json · Qwen3MoeForCausalLM · model repo

Type: Sparse MoE decoder (Qwen3-235B-A22B): 128 experts, GQA
Attention: Grouped-Query Attention: 64 Q heads / 4 KV heads, head dim 128, RoPE theta 1M; no SWA
Decoder: MoE causal decoder-only (94 layers, decoder_sparse_step 1)
MoE: yes (128 experts)
Routing: Top-8 of 128 experts per token, no shared expert

Training Pipeline

  1. 1
    pretraining

    Pre-training (~36T tokens, 119 languages)

    Large-scale multilingual pre-training; 36T tokens.

  2. 2
    sft

    Post-training stage 1-2: SFT + strong-to-weak distillation

    Four-stage post-training pipeline of Qwen3: long CoT cold start, reasoning-centric SFT, thinking-mode fusion, general RL (per Qwen3 blog; summarized as SFT here).

  3. 3
    rl

    Post-training stage 3-4: reasoning RL on prompts + general RL

    Reasoning-centric RL on ~20K prompts plus general-domain RL, completing the four-stage post-training pipeline.

Training & Evaluation Datasets

NameRoleSizeModalitiesCollection
~36T token pretraining corpus (119 languages) pretraining — —

Linked Resources

blog

Qwen3 launch blog (QwenLM)

https://qwenlm.github.io/blog/qwen3/

 github

QwenLM/Qwen3 (GitHub)

https://github.com/QwenLM/Qwen3

 website

Qwen documentation

https://qwen.readthedocs.io/en/latest/

 collection

Qwen3 (HuggingFace collection)

https://huggingface.co/collections/Qwen/qwen3-66dd1b639b283947e839b4ce

 demo

Qwen3 demo (chat.qwen.ai)

https://chat.qwen.ai

 citation

BibTeX citation (Qwen team)

https://huggingface.co/Qwen/Qwen3-235B-A22B

Comments (0)

No comments yet

Be the first to share your thoughts!

What now?

You have read the full card for Qwen3-235B-A22B — here is where to go next.

Compare with another model Test on your own GPU Related models Data as JSON