SenseNova-Loop-9B-Preview

About

SenseNova-Loop-9B-Preview is a multimodal language model developed for long-context and agentic workflows. It accepts text and images and generates text for applications that combine document understanding, coding, and interaction with external tools.

The model combines a looped decoder with two-stream manifold-constrained hyper-connections (mHC): the decoder block of layers 4–27 is executed twice with shared weights (56 decoder-layer executions per forward pass from 32 distinct layers), while learned, Sinkhorn-normalized mappings (20 iterations, FP32 routing parameters) manage information flow across two residual streams inside the looped section. Attention is hybrid, interleaving three linear-attention layers with one full-attention layer in a repeating four-layer pattern.

Key highlights (from the model card):

• Shared looped block: [4, 28) applied 2x with shared weights
• Two-stream mHC: streams = 2, 20 Sinkhorn iterations
• Hybrid attention: 3 linear-attention + 1 full-attention layer per 4-layer group
• Long-context multimodal training: text, images, code, and tool trajectories at sequence lengths up to 262,144 tokens
• MTP available for accelerated inference; DSpark and DFlash2 draft models in training

Backbone scale ~9B total parameters (~14.6B inference parameters), hidden size 4,096, FFN 12,288, vocabulary 248,320, 16 attention heads / 4 KV heads. Deployment documented via Transformers (trust_remote_code), LightLLM, SGLang (PR #47281) and vLLM (PR #64832).

Training Data Documented training recipes combine long sequences, multimodal inputs, and supervised task trajectories, with a maximum training sequence length of 262,144 tokens (model card, Overview/Training).