SmolVLM2-256M-Video-Instruct

HuggingFaceTB

SmolVLM (Idefics3-based vision-language model: SigLIP vision encoder + Llama-3-style text decoder)

Open Weights Commercial Use Multimodal F32 SmolVLM English

Parameters

256.5M

Context

8,192 tokens

native

Modalities

text, image, video text

input → output

Benchmarks

#134/135

better than 1% of all scored models

License

Apache License 2.0

About

SmolVLM2-256M-Video is a lightweight multimodal model designed to analyze video content. The model processes videos, images, and text inputs to generate text outputs — answering questions about media files, comparing visual content, or transcribing text from images. Despite its compact size, it requires only 1.38GB of GPU RAM for video inference, making it particularly well-suited for on-device applications with limited computational resources.

Built upon SigLIP as image encoder and SmolLM2 for the text decoder part, following the Idefics3 architecture. Supports captioning, visual question answering and storytelling via arbitrarily interleaved text/image/video inputs; it does not support image or video generation.

  • Model type: Multi-modal (image/multi-image/video/text)
  • Language(s): English
  • License: Apache 2.0
  • Context: 8192 tokens (config.json text_config.max_position_embeddings)
  • Hardware footprint: 1.38GB GPU RAM for video inference

Evaluation per model card (scientific video benchmarks, 256M row): Video-MME 33.7, MLVU 40.6, MVBench 32.7. Trained on 3.3M samples from ten datasets (LLaVA OneVision, M4-Instruct, Mammoth, LLaVA Video 178K, FineVideo, VideoStar, VRipt, Vista-400K, MovieChat, ShareGPT4Video).

Training Data Trained on 3.3M samples from ten datasets: LLaVA OneVision, M4-Instruct, Mammoth (MAmmoTH-VL-Instruct-12M), LLaVA Video 178K, FineVideo, VideoStar (Video-STaR), VRipt, Vista-400K, MovieChat and ShareGPT4Video. Data split per modality (model card): Image 34.4%, Text 20.2%, Video 33.0%, Multi-image 12.3%.

Benchmark Scores

Benchmark Score Date Standing
MLVU
video_understanding
40.60
—
#14/14
better than 0%
MVBench
video_understanding
32.70
—
#13/13
better than 0%

Architecture

Decoder Block input Embedding Full Attention Dense FFN Final RMSNorm LM Head output

Vision-language transformer (SmolVLM/Idefics3 style), dense

Vision
Parameters
256.5M
Attention
GQA Grouped Query Attention
Context
8192
F32

Source: extracted model record · model repo

Training & Evaluation Datasets

NameRoleSizeModalitiesCollection
LLaVA-OneVision-Data training Part of 3.3M samples total across ten datasets mixed
M4-Instruct-Data training — mixed
MAmmoTH-VL-Instruct-12M (Mammoth) training — mixed
LLaVA-Video-178K training — mixed
FineVideo training — mixed
Video-STaR (VideoStar) training — mixed
Vript training — mixed
VISTA-400K training — mixed
MovieChat-1K_train training — mixed
ShareGPT4Video training — mixed

Comments (0)

No comments yet

Be the first to share your thoughts!

What now?

You have read the full card for SmolVLM2-256M-Video-Instruct — here is where to go next.