Gemma-4-31B

About

Gemma-4-31B (google/gemma-4-31B) is the largest dense model of the Gemma 4 family of open models built by Google DeepMind - multimodal models handling text and image input (audio on the E2B/E4B/12B siblings) and generating text, with multilingual support in over 140 languages. The Gemma 4 release includes both pre-trained and instruction-tuned variants in five sizes (E2B, E4B, 12B, 26B A4B MoE, 31B), deployable from high-end phones to laptops and servers.

Its hybrid attention mechanism interleaves local Sliding-Window Attention (SWA) with full global attention (50 sliding layers with window 1024 and 10 global layers in the 31B; the final layer is always global), delivering the speed and low memory footprint of a lightweight model without sacrificing long-context awareness. To optimize long-context memory, global layers use unified Keys and Values (K=V, 4 global KV heads, head dim 512) and apply Proportional RoPE (p-RoPE) (theta 1M, partial rotary 0.25 on global layers; 10k on sliding). The 31B has 30.7B parameters, 60 layers, a 262,144-token context window, 262K vocabulary, and a ~550M vision encoder (27-layer ViT, 280 soft tokens per image, variable aspect ratio and resolution).

Gemma 4 introduces configurable thinking modes, native function calling for agentic workflows, native system role support, and enhanced coding capability. Pre-training data spans web documents, code, mathematics and images (cutoff January 2025) in over 140 languages, with CSAM and sensitive-data filtering. Smaller siblings use Per-Layer Embeddings (PLE) for parameter efficiency; the 12B "Unified" sibling is encoder-free. Released March 12, 2026 under Apache 2.0.

Training Data Pre-trained on a large-scale multimodal corpus (web documents, code, mathematics, images; 140+ languages; data cutoff January 2025) with CSAM and sensitive-data filtering. Instruction-tuned variants released separately.