Muse-Spark-Open-816B-A42B

About

Muse Spark Open (meta-models/Muse-Spark-Open-816B-A42B) is Meta Superintelligence Labs' 816B-total / 42B-active parameter open-weights Mixture-of-Experts Transformer with a dedicated Perception Encoder, released October 15, 2026 under the Muse Model License. The language model has 62 layers, hidden dimension 8192, Gated Attention (output gate and output norm on the attention block, QK-norm enabled) with Grouped-Query Attention (128 Q heads / 16 KV heads, head dimension 64), untied input/output embeddings (202,048 × 8192 ≈ 1.66B params each), and 256 routed experts with top-8 routing per token (42B active parameters). It processes a 1M-token context window at inference and is natively multimodal: text, image, video and audio input, text output, trained on data from more than 100 languages. The 1.8B-parameter Perception Encoder (50 layers, 1536 hidden dim, 16 heads, 14×14 patches, 4096 output dim) uses learned absolute + 2D RoPE positional embeddings and sparse attention with factor 4 (every 4th and the last layer are full attention).

Built for coding and agentic work, Muse Spark Open covers multi-step reasoning over long horizons, end-to-end agentic task completion (multi-step planning, sequential tool invocation, failure recovery, long-horizon execution within scaffolds), coding agents, tool use with diagnose-and-retry behavior, multimodal reasoning over screenshots, charts, documents, images, audio and video, synthetic data generation, and LLM-as-a-judge evaluation. The open-weights version has additional safety post-training for standalone use without system guardrails; Meta recommends deploying it as part of a broader AI system with guardrails suited to the use case. Recommended inference parameters: temperature 1.0, top_p 1.0, top_k disabled, with controllable Reasoning Effort (low / medium / high / xhigh, default xhigh) set via the reasoning_effort request parameter. Weights ship in BF16 and NVFP4 versions: NVFP4 requires a GPU with at least 700GB of VRAM, BF16 2TB; supported frameworks are SGLang, vLLM, Hugging Face, TensorRT-LLM, Unsloth and llama.cpp, plus MLX on two Mac Studios with M5 Ultra.

Training Data Trained on text, images, audio, and video from public sources, Internet crawls, third-party acquisitions, Meta's products and services, and synthetic data curated and enriched by external vendor networks and Meta personnel; curation includes cleaning, deduplication, and filtering. Multilingual data from more than 100 languages.