Gemma4-26B-A4B

Google

Parameters

25.2B total / 3.8B active

MoE: total / active

Architecture

Mixture-of-Experts (MoE)

Released

11.03.2026

License

Apache License 2.0

Open Weights Commercial Use Multimodal BF16 Gemma4 en multilingual

Input Modalities

text image

Output Modalities

text

Context (native)

262,144 tokens

Context (extended)

262,144 tokens

Openness Index Score 100.0/100

About

Gemma4-26B-A4B (google/gemma-4-26B-A4B-it) is the instruction-tuned Mixture-of-Experts model of Google DeepMind's Gemma 4 family - a sparse model handling text and image input and generating text, with multilingual support in over 140 languages. The "A" stands for "active parameters": of its 25.2B total parameters (128 routed experts plus 1 shared expert), only a 3.8B subset activates per token (top-8 routing), so it runs almost as fast as a 4B-parameter model - an excellent choice for fast inference compared to the dense 31B sibling.

Its 30-layer hybrid attention stack interleaves local Sliding-Window Attention (SWA) (25 layers, window 1024) with 5 full-attention layers (final layer global), with unified Keys and Values (K=V, 2 global KV heads, head dim 512) and Proportional RoPE (p-RoPE) on global layers, 16 Q / 8 KV heads (head dim 256), hidden size 2816, dense FFN 2112, expert FFN 704, 262,144-token context window and 262K vocabulary, with soft logit capping at 30.0. Multimodality comes from a ~550M vision encoder (27-layer ViT, patch 16, 280 soft tokens per image, variable aspect ratio and resolution); audio is not supported on this variant.

Gemma 4 capabilities include configurable thinking modes, native function calling for agentic workflows, native system role support, video understanding via frame sequences, interleaved multimodal input and enhanced coding. Pre-training spans web documents, code, mathematics and images (cutoff January 2025, 140+ languages) with CSAM and sensitive-data filtering; this checkpoint is the instruction-tuned variant. Released March 11, 2026 under Apache 2.0.

Training Data Large-scale diverse collection: web documents, code, images, audio. 140+ languages. Cutoff: January 2025.

Benchmark Scores

Benchmark Score Date Standing
SWE-bench Verified
coding_agent
19.50%
16.04.2026
#54/60
better than 10%
SWE-bench Multilingual
coding_agent
14.56%
16.04.2026
#47/48
better than 2%
Terminal-Bench 2.0
coding_agent
6.28%
16.04.2026
#21/23
better than 9%
Claw-Eval Avg
coding_agent
70.39%
16.04.2026
#23/40
better than 44%
SkillsBench Avg5
coding_agent
12.68%
16.04.2026
#15/16
better than 7%
QwenClawBench
coding_agent
69.94%
16.04.2026
#9/17
better than 50%
NL2Repo
coding_agent
2.00%
16.04.2026
#40/41
better than 2%
QwenWebBench
coding_agent
35.84%
16.04.2026
#6/8
better than 29%
TAU3-Bench
general_agent
85.38%
16.04.2026
#7/21
better than 70%
VITA-Bench
general_agent
84.20%
16.04.2026
#3/16
better than 87%

Showing 10 of 62 results

What is a good SWE-bench Verified score? What is a good SWE-bench Multilingual score? What is a good Terminal-Bench 2.0 score?

Hardware Requirements

Estimated VRAM per quantization at 4,096 tokens context — MoE: all experts must fit in VRAM, only 3.8B are active per token

Quantization Weights KV Cache Total VRAM On your hardware
FP16 — 16-bit 50.4 GB 0.6 GB 53.5 GB Which GPUs can run it? Calculate on your GPU
INT8 — 8-bit 25.2 GB 0.3 GB 26.8 GB Which GPUs can run it? Calculate on your GPU
INT4 — 4-bit 12.6 GB 0.1 GB 13.4 GB Which GPUs can run it? Calculate on your GPU
FP4 — 4-bit 12.6 GB 0.1 GB 13.4 GB Which GPUs can run it? Calculate on your GPU

Consumer GPUs that fit this model with a single card (INT4):

Apple M3 Pro · 18GB AMD Radeon RX 7900 XT · 20GB GeForce RTX 3090 · 24GB GeForce RTX 4090 D · 24GB AMD Radeon RX 7900 XTX · 24GB GeForce RTX 3090 Ti · 24GB GeForce RTX 4090 · 24GB Apple M4 Pro · 24GB Apple M1 Max · 32GB Apple M2 Max · 32GB GeForce RTX 5090 · 32GB Apple M3 Max · 36GB

Architecture

Decoder Block ×30 input Embedding vocab 262K · d 2816 Sliding Window Attn Hybrid 16:8 · dₕ 256 · win 1024 ×25 Full Attention Hybrid 16:8 · dₕ 256 · win 1024 ×5 MoE FFN 128 experts · dᴻ 704 Final RMSNorm LM Head tied with embedding output
Attention
Hybrid Attention (16:8)
MoE
128 experts
Layers
30
Hidden size
2816
Context
262K tokens
Parameters
25200M
Active params
3800M

Source: Hugging Face config.json · Gemma4ForConditionalGeneration · exact layer pattern · model repo

Type: Sparse MoE hybrid-attention (Gemma 4 26B-A4B): 25 SWA + 5 full-attn layers
Attention: Interleaved local SWA (window 1024; 16 Q / 8 KV GQA) with 5 full-attention layers; unified K=V (2 global KV heads, head dim 512); Proportional RoPE theta 1M partial 0.25 on global / 10k on sliding; softcap 30.0
Decoder: Sparse MoE decoder-only hybrid (SWA + full attention)
MoE: yes (128 experts)
Routing: Top-8 of 128 routed experts + 1 shared expert

Training Pipeline

  1. 1
    pretraining

    Multimodal pre-training (cutoff January 2025)

    Web documents, code, mathematics and images in 140+ languages; CSAM and sensitive-data filtering.

  2. 2
    sft

    Instruction tuning

    This checkpoint is the instruction-tuned (-it) variant of Gemma 4 26B A4B.

Training & Evaluation Datasets

NameRoleSizeModalitiesCollection
Web documents (140+ languages) pretraining — —
Code pretraining — —
Mathematics pretraining — —
Images pretraining — —

Linked Resources

paper

Gemma 4 Technical Report

https://arxiv.org/abs/2607.02770

 blog

Gemma 4 launch blog (Google)

https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/

 website

Gemma documentation (Google AI for Developers)

https://ai.google.dev/gemma/docs/core

 github

google-gemma (GitHub)

https://github.com/google-gemma

 collection

Gemma 4 (HuggingFace collection)

https://huggingface.co/collections/google/gemma-4

 citation

BibTeX citation (Google DeepMind, 2026)

https://huggingface.co/google/gemma-4-26B-A4B-it

Comments (0)

No comments yet

Be the first to share your thoughts!

What now?

You have read the full card for Gemma4-26B-A4B — here is where to go next.

Compare with another model Test on your own GPU Related models Data as JSON