Parameters
12.0B
Architecture
Transformer (decoder-only)
Released
27.02.2025
License
Gemma Terms of Use
Input Modalities
Output Modalities
Context (native)
131,072 tokens
Context (extended)
131,072 tokens
About
Gemma 3 12B PT is the pre-trained-variant record for Google's Gemma 3 12B model - a lightweight, state-of-the-art open multimodal transformer (12B parameters, dense) built from the same research and technology used to create the Gemini models, released February 27, 2025 under the Gemma Terms of Use. The underlying google/gemma-3-12b repository is access-gated; this record carries the pre-training (PT) benchmark scores for the 12B model.
Gemma 3 handles text and image input and generates text output, with a 128K context window and multilingual support in over 140 languages. The 12B model was trained on 12 trillion tokens of web documents, code, mathematics and images. Its architecture interleaves local sliding-window and global attention layers in a 5:1 local-global pattern, applies QK-Norm for stable attention, and uses a SigLIP-based vision encoder for image understanding.
Training Data 12 trillion tokens (web documents, code, mathematics, images) in 140+ languages
Benchmark Scores
| Benchmark | Score | Date | Standing |
|---|---|---|---|
|
BoolQ
reasoning
|
81.25%
|
27.02.2025 |
#2/4
better than 67%
|
|
PIQA
reasoning
|
84.21%
|
27.02.2025 |
#2/4
better than 67%
|
|
SocialIQA
reasoning
|
75.00%
|
27.02.2025 |
#2/4
better than 67%
|
|
TriviaQA
knowledge
|
84.03%
|
27.02.2025 |
#2/4
better than 67%
|
|
Natural Questions
knowledge
|
82.34%
|
27.02.2025 |
#2/4
better than 67%
|
|
BIG-Bench Hard
reasoning
|
75.94%
|
27.02.2025 |
#5/8
better than 43%
|
|
DROP
reasoning
|
85.63%
|
27.02.2025 |
#2/4
better than 67%
|
|
MMLU Pro COT
knowledge
|
31.69%
|
27.02.2025 |
#3/4
better than 33%
|
|
AGIEval
reasoning
|
63.49%
|
27.02.2025 |
#2/3
better than 50%
|
|
MATH
math
|
74.03%
|
27.02.2025 |
#2/3
better than 50%
|
|
GSM8K
math
|
55.96%
|
27.02.2025 |
#4/6
better than 40%
|
|
GPQA
reasoning
|
16.72%
|
27.02.2025 |
#12/14
better than 15%
|
|
MBPP
code
|
73.47%
|
27.02.2025 |
#2/3
better than 50%
|
|
HumanEval
code
|
75.78%
|
27.02.2025 |
#2/3
better than 50%
|
|
MGSM
multilingual
|
73.28%
|
27.02.2025 |
#3/5
better than 50%
|
|
Global-MMLU-Lite
multilingual
|
87.60%
|
27.02.2025 |
#2/4
better than 67%
|
|
WMT24++ (en→xx)
multilingual
|
34.33%
|
27.02.2025 |
#15/17
better than 12%
|
|
FloRes
multilingual
|
85.49%
|
27.02.2025 |
#2/4
better than 67%
|
|
XQuAD
multilingual
|
93.01%
|
27.02.2025 |
#2/4
better than 67%
|
|
ECLeKTic
multilingual
|
63.47%
|
27.02.2025 |
#2/4
better than 67%
|
|
IndicGenBench
multilingual
|
92.27%
|
27.02.2025 |
#2/4
better than 67%
|
|
COCOcap
multimodal
|
64.29%
|
27.02.2025 |
#2/3
better than 50%
|
|
DocVQA
multimodal
|
39.75%
|
27.02.2025 |
#4/5
better than 25%
|
|
MMMU
vision_language
|
24.24%
|
27.02.2025 |
#17/19
better than 11%
|
|
RealWorldQA
vision_language
|
15.58%
|
27.02.2025 |
#22/23
better than 5%
|
|
ReMI
multimodal
|
64.00%
|
27.02.2025 |
#2/3
better than 50%
|
|
AI2D_TEST
document_understanding
|
39.87%
|
27.02.2025 |
#14/15
better than 7%
|
|
ChartQA
multimodal
|
87.40%
|
27.02.2025 |
#2/3
better than 50%
|
|
BLINK
multimodal
|
35.90
|
27.02.2025 |
#5/5
better than 0%
|
|
OKVQA
multimodal
|
83.70%
|
27.02.2025 |
#2/3
better than 50%
|
|
TallyQA
multimodal
|
78.81%
|
27.02.2025 |
#2/3
better than 50%
|
|
SpatialSense VQA
multimodal
|
100.00%
|
27.02.2025 |
#1/3
better than 100%
|
|
CountBenchQA
multimodal
|
17.80
|
27.02.2025 |
#4/4
better than 0%
|
|
HellaSwag
reasoning
|
60.33%
|
27.02.2025 |
#5/7
better than 33%
|
|
ARC-c
reasoning
|
50.92%
|
27.02.2025 |
#4/6
better than 40%
|
|
WinoGrande
reasoning
|
40.97%
|
27.02.2025 |
#4/6
better than 40%
|
|
MMLU
knowledge
|
44.21%
|
27.02.2025 |
#16/17
better than 6%
|
|
ARC-e
reasoning
|
57.74%
|
27.02.2025 |
#4/6
better than 40%
|
|
TextVQA
multimodal
|
26.21%
|
27.02.2025 |
#3/4
better than 33%
|
|
VQAv2
multimodal
|
25.35%
|
27.02.2025 |
#3/4
better than 33%
|
|
InfoVQA
multimodal
|
25.42%
|
27.02.2025 |
#4/5
better than 25%
|
What is a good BoolQ score? What is a good PIQA score? What is a good SocialIQA score?
Hardware Requirements
Estimated VRAM per quantization at 4,096 tokens context
| Quantization | Weights | KV Cache | Total VRAM | On your hardware |
|---|---|---|---|---|
| FP16 — 16-bit | 24.0 GB | 1.9 GB | 27.1 GB | Which GPUs can run it? Calculate on your GPU |
| INT8 — 8-bit | 12.0 GB | 0.9 GB | 13.5 GB | Which GPUs can run it? Calculate on your GPU |
| INT4 — 4-bit | 6.0 GB | 0.5 GB | 6.8 GB | Which GPUs can run it? Calculate on your GPU |
| FP4 — 4-bit | 6.0 GB | 0.5 GB | 6.8 GB | Which GPUs can run it? Calculate on your GPU |
Consumer GPUs that fit this model with a single card (INT4):
Architecture
- Attention
- Grouped Query Attention
- Context
- 131K tokens
- Parameters
- 12000M
Source: extracted model record · model repo
Training Pipeline
-
1
pretraining
Pretraining on 12 trillion tokens
Pretrained on 12 trillion tokens of web documents, code, mathematics, and images in 140+ languages. Trained on TPU (TPUv4p, TPUv5p, TPUv5e) using JAX and ML Pathways.
Training & Evaluation Datasets
| Name | Role | Size | Modalities | Collection |
|---|---|---|---|---|
| Web Documents (Multilingual) | pretraining | 14T tokens (27B), 12T (12B), 4T (4B), 2T (1B) | text | automated |
| Code Training Data | pretraining | — | code | automated |
| Mathematics Training Data | pretraining | — | text | automated |
| Image Training Data | pretraining | — | image | automated |
| CSAM-Filtered Data | pretraining | — | automated | |
| Sensitive Data Filtered | pretraining | — | automated |
Linked Resources
Gemma 3 Technical Report
https://goo.gle/Gemma3Report
Gemma Model Page
https://ai.google.dev/gemma/docs/core
Responsible Generative AI Toolkit
https://ai.google.dev/responsible
Gemma on Kaggle
https://www.kaggle.com/models/google/gemma-3
Gemma on Vertex Model Garden
https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemma3
Gemma Terms of Use
https://ai.google.dev/gemma/terms
Gemma Prohibited Use Policy
https://ai.google.dev/gemma/prohibited_use_policy
Gemini Family Technical Report
https://arxiv.org/abs/2312.11805
ECLeKTic Paper
https://huggingface.co/papers/2502.21228
WMT24++ Paper
https://huggingface.co/papers/2502.12404
IndicGenBench Paper
https://huggingface.co/papers/2404.16816
BLINK Paper
https://huggingface.co/papers/2404.12390
Google Sustainability Commitments
https://sustainability.google/operating-sustainably/
Google AI Responsibility Update
https://ai.google/static/documents/ai-responsibility-update-published-february-2025.pdf
JAX (training framework)
https://github.com/jax-ml/jax
ML Pathways
https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/
Google Cloud TPU
https://cloud.google.com/tpu/docs/intro-to-tpu
Google Gemma Models Family Collection
https://huggingface.co/collections/google/googles-gemma-models-family
Gemma 3 Release Collection
https://huggingface.co/collections/google/gemma-3-release
HuggingFace Inference Providers - Gemma 3 27B IT
https://huggingface.co/inference/models?model=google%2Fgemma-3-27b-it
Trend Analysis
24h Change
+2.4%
7d Change
+6.6%
30d Change
-24.9%
Current
383,043
followers
+0.2%
likes
+0.0%
downloads_all_time
+0.1%
downloads
+0.0%
No comments yet
Be the first to share your thoughts!