Gemma4-12B

About

Gemma4-12B (google/gemma-4-12B-it) is the instruction-tuned 12B "Unified" model of Google DeepMind's Gemma 4 family - an encoder-free multimodal architecture that eliminates dedicated vision and audio encoders entirely: raw image patches and audio waveforms are projected directly into the LLM's embedding space through lightweight linear layers, so all modalities flow into a single decoder-only transformer. It handles text, image and audio input and generates text (any-to-any pipeline), with multilingual support in over 140 languages.

The 12B Unified has 11.95B parameters, 48 layers (40 Sliding-Window Attention (SWA) layers with window 1024 + 8 full-attention layers), hidden size 3840, FFN 15360, 262,144-token context window and 262K vocabulary. Like its siblings it uses the hybrid local-global attention design (final layer always global), unified Keys and Values (K=V) with Proportional RoPE (p-RoPE) on global layers (1 global KV head, head dim 512, theta 1M partial rotary 0.25), and final logit softcapping at 30.0.

Gemma 4 capabilities include configurable thinking modes, native function calling for agentic workflows, native system role support, video understanding via frame sequences, interleaved multimodal input, and enhanced coding. Pre-training spans web documents, code, mathematics, images and audio (cutoff January 2025, 140+ languages) with CSAM and sensitive-data filtering; this checkpoint is the instruction-tuned variant. Released May 23, 2026 under Apache 2.0.

Training Data Large-scale diverse collection: web documents, code, images, audio. 140+ languages. Cutoff: January 2025.