SmolVLM2-256M-Video-Instruct
SmolVLM (Idefics3-based vision-language model: SigLIP vision encoder + Llama-3-style text decoder)
Parameters
256.5M
Context
8,192 tokens
native
Modalities
text, image, video text
input → output
Benchmarks
#134/135
better than 1% of all scored models
License
Apache License 2.0
About
SmolVLM2-256M-Video is a lightweight multimodal model designed to analyze video content. The model processes videos, images, and text inputs to generate text outputs — answering questions about media files, comparing visual content, or transcribing text from images. Despite its compact size, it requires only 1.38GB of GPU RAM for video inference, making it particularly well-suited for on-device applications with limited computational resources.
Built upon SigLIP as image encoder and SmolLM2 for the text decoder part, following the Idefics3 architecture. Supports captioning, visual question answering and storytelling via arbitrarily interleaved text/image/video inputs; it does not support image or video generation.
- Model type: Multi-modal (image/multi-image/video/text)
- Language(s): English
- License: Apache 2.0
- Context: 8192 tokens (config.json
text_config.max_position_embeddings) - Hardware footprint: 1.38GB GPU RAM for video inference
Evaluation per model card (scientific video benchmarks, 256M row): Video-MME 33.7, MLVU 40.6, MVBench 32.7. Trained on 3.3M samples from ten datasets (LLaVA OneVision, M4-Instruct, Mammoth, LLaVA Video 178K, FineVideo, VideoStar, VRipt, Vista-400K, MovieChat, ShareGPT4Video).
Training Data Trained on 3.3M samples from ten datasets: LLaVA OneVision, M4-Instruct, Mammoth (MAmmoTH-VL-Instruct-12M), LLaVA Video 178K, FineVideo, VideoStar (Video-STaR), VRipt, Vista-400K, MovieChat and ShareGPT4Video. Data split per modality (model card): Image 34.4%, Text 20.2%, Video 33.0%, Multi-image 12.3%.
Benchmark Scores
| Benchmark | Score | Date | Standing |
|---|---|---|---|
|
MLVU
video_understanding
|
40.60
|
— |
#14/14
better than 0%
|
|
MVBench
video_understanding
|
32.70
|
— |
#13/13
better than 0%
|
Hardware Requirements
Estimated VRAM per quantization at 4,096 tokens context
| Quantization | Weights | KV Cache | Total VRAM | On your hardware |
|---|---|---|---|---|
| FP16 — 16-bit | 0.5 GB | 0.0 GB | 0.6 GB | Which GPUs can run it? Calculate on your GPU |
| INT8 — 8-bit | 0.3 GB | 0.0 GB | 0.3 GB | Which GPUs can run it? Calculate on your GPU |
| INT4 — 4-bit | 0.1 GB | 0.0 GB | 0.1 GB | Which GPUs can run it? Calculate on your GPU |
| FP4 — 4-bit | 0.1 GB | 0.0 GB | 0.1 GB | Which GPUs can run it? Calculate on your GPU |
Consumer GPUs that fit this model with a single card (INT4):
Architecture
Vision-language transformer (SmolVLM/Idefics3 style), dense
- Parameters
- 256.5M
- Attention
- GQA Grouped Query Attention
- Context
- 8192
Source: extracted model record · model repo
Training & Evaluation Datasets
| Name | Role | Size | Modalities | Collection |
|---|---|---|---|---|
| LLaVA-OneVision-Data | training | Part of 3.3M samples total across ten datasets | mixed | |
| M4-Instruct-Data | training | — | mixed | |
| MAmmoTH-VL-Instruct-12M (Mammoth) | training | — | mixed | |
| LLaVA-Video-178K | training | — | mixed | |
| FineVideo | training | — | mixed | |
| Video-STaR (VideoStar) | training | — | mixed | |
| Vript | training | — | mixed | |
| VISTA-400K | training | — | mixed | |
| MovieChat-1K_train | training | — | mixed | |
| ShareGPT4Video | training | — | mixed |
Linked Resources
SmolVLM2 blog post
https://huggingface.co/blog/smolvlm2
SmolVLM2 Video Highlight Generator (Demo Space)
https://huggingface.co/spaces/HuggingFaceTB/SmolVLM2-HighlightGenerator
SmolVLM: Redefining small and efficient multimodal models (arXiv 2504.05299)
https://arxiv.org/abs/2504.05299
Fine-tuning tutorial (Smol_VLM_FT.ipynb, huggingface/smollm)
https://github.com/huggingface/smollm/blob/main/vision/finetuning/Smol_VLM_FT.ipynb
No comments yet
Be the first to share your thoughts!