Parameters
4.0B
Architecture
Transformer (decoder-only)
Released
27.02.2025
License
Gemma Terms of Use
Input Modalities
Output Modalities
Context (native)
131,072 tokens
Context (extended)
131,072 tokens
About
Gemma 3 4B PT is the pre-trained-variant record for Google's Gemma 3 4B model - a lightweight, open multimodal transformer (4B parameters, dense) built from the same research and technology used to create the Gemini models, released February 27, 2025 under the Gemma Terms of Use. The underlying google/gemma-3-4b repository is access-gated; this record carries the pre-training (PT) benchmark scores for the 4B model.
Gemma 3 handles text and image input and generates text output, with a 128K context window and multilingual support in over 140 languages. The 4B model was trained on 4 trillion tokens of web documents, code, mathematics and images. Its architecture interleaves local sliding-window and global attention layers in a 5:1 local-global pattern, applies QK-Norm for stable attention, and uses a SigLIP-based vision encoder for image understanding.
Training Data 4 trillion tokens (web documents, code, mathematics, images) in 140+ languages
Benchmark Scores
| Benchmark | Score | Date | Standing |
|---|---|---|---|
|
HellaSwag
reasoning
|
41.05%
|
27.02.2025 |
#6/7
better than 17%
|
|
BoolQ
reasoning
|
47.40%
|
27.02.2025 |
#3/4
better than 33%
|
|
PIQA
reasoning
|
61.05%
|
27.02.2025 |
#3/4
better than 33%
|
|
SocialIQA
reasoning
|
50.00%
|
27.02.2025 |
#3/4
better than 33%
|
|
TriviaQA
knowledge
|
56.89%
|
27.02.2025 |
#3/4
better than 33%
|
|
Natural Questions
knowledge
|
39.52%
|
27.02.2025 |
#3/4
better than 33%
|
|
BIG-Bench Hard
reasoning
|
38.66%
|
27.02.2025 |
#7/8
better than 14%
|
|
DROP
reasoning
|
50.86%
|
27.02.2025 |
#3/4
better than 33%
|
|
MMLU
knowledge
|
59.60
|
27.02.2025 |
#17/17
better than 0%
|
|
MMLU Pro COT
knowledge
|
29.20
|
27.02.2025 |
#4/4
better than 0%
|
|
AGIEval
reasoning
|
42.10
|
27.02.2025 |
#3/3
better than 0%
|
|
MATH
math
|
24.20
|
27.02.2025 |
#3/3
better than 0%
|
|
GSM8K
math
|
38.40
|
27.02.2025 |
#6/6
better than 0%
|
|
GPQA
reasoning
|
15.00
|
27.02.2025 |
#14/14
better than 0%
|
|
HumanEval
code
|
36.00
|
27.02.2025 |
#3/3
better than 0%
|
|
MGSM
multilingual
|
38.44%
|
27.02.2025 |
#4/5
better than 25%
|
|
Global-MMLU-Lite
multilingual
|
63.19%
|
27.02.2025 |
#3/4
better than 33%
|
|
WMT24++ (en→xx)
multilingual
|
23.35%
|
27.02.2025 |
#16/17
better than 6%
|
|
FloRes
multilingual
|
50.26%
|
27.02.2025 |
#3/4
better than 33%
|
|
XQuAD
multilingual
|
73.25%
|
27.02.2025 |
#3/4
better than 33%
|
|
ECLeKTic
multilingual
|
32.01%
|
27.02.2025 |
#3/4
better than 33%
|
|
IndicGenBench
multilingual
|
71.82%
|
27.02.2025 |
#3/4
better than 33%
|
|
COCOcap
multimodal
|
102.00
|
27.02.2025 |
#3/3
better than 0%
|
|
DocVQA
multimodal
|
72.80
|
27.02.2025 |
#5/5
better than 0%
|
|
InfoVQA
multimodal
|
44.10
|
27.02.2025 |
#5/5
better than 0%
|
|
MMMU
vision_language
|
39.20
|
27.02.2025 |
#19/19
better than 0%
|
|
TextVQA
multimodal
|
58.90
|
27.02.2025 |
#4/4
better than 0%
|
|
RealWorldQA
vision_language
|
45.50
|
27.02.2025 |
#23/23
better than 0%
|
|
ReMI
multimodal
|
27.30
|
27.02.2025 |
#3/3
better than 0%
|
|
AI2D_TEST
document_understanding
|
63.20
|
27.02.2025 |
#15/15
better than 0%
|
|
ChartQA
multimodal
|
63.60
|
27.02.2025 |
#3/3
better than 0%
|
|
VQAv2
multimodal
|
63.90
|
27.02.2025 |
#4/4
better than 0%
|
|
OKVQA
multimodal
|
51.00
|
27.02.2025 |
#3/3
better than 0%
|
|
TallyQA
multimodal
|
42.50
|
27.02.2025 |
#3/3
better than 0%
|
|
SpatialSense VQA
multimodal
|
50.90
|
27.02.2025 |
#3/3
better than 0%
|
|
BLINK
multimodal
|
5.97%
|
27.02.2025 |
#4/5
better than 25%
|
|
MBPP
code
|
46.00
|
27.02.2025 |
#3/3
better than 0%
|
|
ARC-c
reasoning
|
29.72%
|
27.02.2025 |
#5/6
better than 20%
|
|
WinoGrande
reasoning
|
16.54%
|
27.02.2025 |
#5/6
better than 20%
|
|
ARC-e
reasoning
|
35.47%
|
27.02.2025 |
#5/6
better than 20%
|
|
CountBenchQA
multimodal
|
11.04%
|
27.02.2025 |
#3/4
better than 33%
|
What is a good HellaSwag score? What is a good BoolQ score? What is a good PIQA score?
Hardware Requirements
Estimated VRAM per quantization at 4,096 tokens context
| Quantization | Weights | KV Cache | Total VRAM | On your hardware |
|---|---|---|---|---|
| FP16 — 16-bit | 8.0 GB | 0.6 GB | 9.0 GB | Which GPUs can run it? Calculate on your GPU |
| INT8 — 8-bit | 4.0 GB | 0.3 GB | 4.5 GB | Which GPUs can run it? Calculate on your GPU |
| INT4 — 4-bit | 2.0 GB | 0.2 GB | 2.3 GB | Which GPUs can run it? Calculate on your GPU |
| FP4 — 4-bit | 2.0 GB | 0.2 GB | 2.3 GB | Which GPUs can run it? Calculate on your GPU |
Consumer GPUs that fit this model with a single card (INT4):
Architecture
- Attention
- Grouped Query Attention
- Context
- 131K tokens
- Parameters
- 4000M
Source: extracted model record · model repo
Training Pipeline
-
1
pretraining
Pretraining on 4 trillion tokens
Pretrained on 4 trillion tokens of web documents, code, mathematics, and images in 140+ languages. Trained on TPU (TPUv4p, TPUv5p, TPUv5e) using JAX and ML Pathways.
Training & Evaluation Datasets
| Name | Role | Size | Modalities | Collection |
|---|---|---|---|---|
| Web Documents (Multilingual) | pretraining | 14T tokens (27B), 12T (12B), 4T (4B), 2T (1B) | text | automated |
| Code Training Data | pretraining | — | code | automated |
| Mathematics Training Data | pretraining | — | text | automated |
| Image Training Data | pretraining | — | image | automated |
| CSAM-Filtered Data | pretraining | — | automated | |
| Sensitive Data Filtered | pretraining | — | automated |
Linked Resources
Gemma 3 Technical Report
https://goo.gle/Gemma3Report
Gemma Model Page
https://ai.google.dev/gemma/docs/core
Responsible Generative AI Toolkit
https://ai.google.dev/responsible
Gemma on Kaggle
https://www.kaggle.com/models/google/gemma-3
Gemma on Vertex Model Garden
https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemma3
Gemma Terms of Use
https://ai.google.dev/gemma/terms
Gemma Prohibited Use Policy
https://ai.google.dev/gemma/prohibited_use_policy
Gemini Family Technical Report
https://arxiv.org/abs/2312.11805
ECLeKTic Paper
https://huggingface.co/papers/2502.21228
WMT24++ Paper
https://huggingface.co/papers/2502.12404
IndicGenBench Paper
https://huggingface.co/papers/2404.16816
BLINK Paper
https://huggingface.co/papers/2404.12390
Google Sustainability Commitments
https://sustainability.google/operating-sustainably/
Google AI Responsibility Update
https://ai.google/static/documents/ai-responsibility-update-published-february-2025.pdf
JAX (training framework)
https://github.com/jax-ml/jax
ML Pathways
https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/
Google Cloud TPU
https://cloud.google.com/tpu/docs/intro-to-tpu
Google Gemma Models Family Collection
https://huggingface.co/collections/google/googles-gemma-models-family
Gemma 3 Release Collection
https://huggingface.co/collections/google/gemma-3-release
HuggingFace Inference Providers - Gemma 3 27B IT
https://huggingface.co/inference/models?model=google%2Fgemma-3-27b-it
Trend Analysis
24h Change
+0.2%
7d Change
+0.9%
30d Change
+4.3%
Current
73,412
likes
+0.1%
downloads
+0.1%
downloads_all_time
+0.1%
downloads
+0.0%
No comments yet
Be the first to share your thoughts!