Raon-1 nano
hybrid dense Transformer decoder with Raon Delta Attention (18 linear attention layers) and Gated MLA (6 full-attention layers), Raon-mHC residual connections (4 streams), per-layer embeddings (256), tied LM head, NoPE
Parameters
3.1B
Context
262,144 tokens
native
Modalities
text text
input → output
Benchmarks
#114/134
better than 15% of all scored models
Size class
#15/19
better than 21% of similar-size models
License
Apache License 2.0
About
Raon-1 nano is a dense language model with 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table), developed by KRAFTON AI. It is designed to follow instructions, maintain conversational context, and call the right tools in both Korean and English. As part of the Raon open model family, it is released under the Apache 2.0 license.
The checkpoint contains 3.2B parameters: 3.1B for the main model and 55M for a Multi-Token Prediction (MTP) module. (The repo safetensors box lists ~3.5B params in BF16/F32.)
Architecture
Hybrid dense architecture: 18 linear attention layers (Raon Delta Attention) + 6 Gated MLA full-attention layers, 24 layers total, hidden size 1,536, FFN intermediate size 6,144, SwiGLU activation, NoPE (no positional encoding), vocabulary of 259,840 with a tied LM head.
- Raon Delta Attention (RDA): gated delta-rule linear attention with separate erase and write gates and a fixed-size recurrent state (no KV cache).
- Raon-mHC: reduces the computational cost of DeepSeek's manifold-constrained hyper-connections (mHC), used with 4 residual streams.
- Per-layer embedding: each layer has its own 256-dimensional embedding per token to adjust its output.
- Only the 6 Gated MLA layers keep a KV cache, about 7 KiB per token in BF16.
- Long context: natively supports up to 262,144 tokens.
Training
Derived from Raon-1 mini preview (30B-A4B) through MoE-to-Dense pruning and knowledge distillation. Continued pre-training on 3T tokens; post-training with supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview, followed by reinforcement learning (RL).
Use and deployment
Built-in thinking mode, enabled by default and recommended to keep on. Tool calling is supported for everyday tasks (office work, mobile control). Recommended sampling: temperature 0.1 / presence penalty 1.5 for general tasks; temperature 1.0 / no penalty for agentic and code. Inference is supported with Transformers, vLLM (KRAFTON fork), llama.cpp (KRAFTON branch), MLX, and LiteRT-LM; GGUF, 4-bit MLX, and LiteRT-LM (4-bit weights, 4,096-token context) bundles are provided for on-device local inference.
Benchmark highlights (model card)
Leads the compared models of similar size on instruction following (IFBench 75.9 vs. 39.4-73.7; MultiChallenge 40.5), context following (FACTS Grounding 44.4), and Korean knowledge/conversation (KMMLU-Redux 50.9, CLIcK 65.2, HAE-RAE 1.1 44.2; internal Chatbot Korean 91.0), with strong tool use (AndroidControl 36.0, NL2SH-ALFA 58.0) and MMLU-Redux 81.1.
Training Data Derived from Raon-1 mini preview (30B-A4B) via MoE-to-Dense pruning and knowledge distillation; continued pre-training on 3T tokens of Korean and English data; post-training with SFT and on-policy distillation (OPD) from Raon-1 mini preview, followed by RL. KRAFTON's proprietary training datasets and methodologies are not distributed with the open release.
Benchmark Scores
| Benchmark | Score | Date | Standing |
|---|---|---|---|
|
MMLU-Redux
knowledge
|
39.08%
|
— |
#25/31
better than 20%
|
|
IFBench
instruction_following
|
86.50%
|
— |
#20/45
better than 57%
|
|
Multi-Challenge
instruction_following
|
40.50
|
— |
#19/19
better than 0%
|
What is a good MMLU-Redux score? What is a good IFBench score? What is a good Multi-Challenge score?
Hardware Requirements
Estimated VRAM per quantization at 4,096 tokens context
| Quantization | Weights | KV Cache | Total VRAM | On your hardware |
|---|---|---|---|---|
| FP16 — 16-bit | 6.2 GB | 0.5 GB | 7.0 GB | Which GPUs can run it? Calculate on your GPU |
| INT8 — 8-bit | 3.1 GB | 0.2 GB | 3.5 GB | Which GPUs can run it? Calculate on your GPU |
| INT4 — 4-bit | 1.6 GB | 0.1 GB | 1.7 GB | Which GPUs can run it? Calculate on your GPU |
| FP4 — 4-bit | 1.6 GB | 0.1 GB | 1.7 GB | Which GPUs can run it? Calculate on your GPU |
Consumer GPUs that fit this model with a single card (INT4):
Architecture
hybrid dense Transformer decoder with Raon Delta Attention (18 linear attention layers) and Gated…
- Parameters
- 3.1B
- Context
- 262K
Training Pipeline
-
1
other
MoE-to-Dense pruning + knowledge distillation from Raon-1 mini preview
Raon-1 nano was derived from Raon-1 mini preview (30B-A4B) through MoE-to-Dense pruning and knowledge distillation (arXiv 2605.28207).
-
2
cpt
Continued pre-training (3T tokens)
Continued pre-training on 3T tokens (Korean and English).
-
3
sft
SFT + on-policy distillation (OPD)
Post-training: supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview.
-
4
rl
Reinforcement learning (RL)
Post-training: reinforcement learning (RL) after SFT/OPD.
Linked Resources
Pruning and Distilling Mixture-of-Experts into Dense Language Models (MoE-to-Dense, arXiv 2605.28207)
https://arxiv.org/abs/2605.28207
vLLM — KRAFTON fork (raon-1-nano branch)
https://github.com/krafton-ai/vllm/tree/raon-1-nano
llama.cpp — KRAFTON branch (raon-1-nano)
https://github.com/krafton-ai/llama.cpp/tree/raon-1-nano
Raon-1 nano collection (on-device open-weight LLM, Apache 2.0)
https://huggingface.co/collections/KRAFTON/raon-1-nano
Raon-1 nano GGUF (llama.cpp)
https://huggingface.co/KRAFTON/Raon-1-nano-gguf
Raon-1 nano MLX (4-bit, Apple silicon)
https://huggingface.co/KRAFTON/Raon-1-nano-mlx
No comments yet
Be the first to share your thoughts!