Kimi K3 is a 2.8 trillion-parameter Mixture-of-Experts model that activates 104 billion per token. That sparsity ratio — roughly 27 to 1 — is the central design decision, and everything else in the architecture exists to make it work at that scale.
The specification
| Component | Detail |
|---|---|
| Total parameters | 2.8 trillion |
| Activated per token | 104 billion |
| Experts | 896 total, 16 selected per token |
| MoE framework | Stable LatentMoE, with latent-space routing and Quantile Balancing |
| Layers | 93 (69 KDA + 24 Gated MLA + 1 dense) |
| Attention | Kimi Delta Attention (KDA) with Gated MLA, plus Attention Residuals |
| Activation | Sigmoid Tanh Unit (SiTU) |
| Optimiser | Per-Head Muon |
| Vocabulary | 160K |
| Context | 1,048,576 tokens tokens |
| Precision | MXFP4 weights, MXFP8 activations |
| Vision encoder | MoonViT-V2, 401M parameters |
Sparsity: 16 of 896 experts
Selecting 16 experts from 896 is aggressive sparsity, and Moonshot reports roughly a 2.5x improvement in scaling efficiency over Kimi K2 as a result.
The practical consequence: compute per token is far below what 2.8T parameters suggests — closer to what a 104B dense model would cost — while capacity is much larger. That is how a model this size can be priced at $3.00 per million input tokens.
The cost is memory. All 2.8T parameters must be resident to serve any token, which is why the weights are ~1.4 TB and why self-hosting is a multi-node problem. Sparsity buys compute, not memory.
Attention built for long context
K3 does not use standard quadratic attention throughout. Of its 93 layers, 69 use Kimi Delta Attention (KDA) — a hybrid linear attention approach — while 24 use Gated MLA and one is dense.
Linear attention scales far better with sequence length, which is the difference between a 1M-token window being a specification and being usable. Attention Residuals complement this by letting later layers retrieve selectively from earlier ones, which is a direct answer to the classic long-context failure of losing material from the middle of a very long input.
Quantisation-aware from the start
The weights ship as MXFP4 (Microscaling FP4, with per-block scaling) and activations as MXFP8. Critically, Moonshot applied quantisation-aware training from the supervised fine-tuning stage onward.
That distinction matters. Most quantised open-weight releases are compressed after training, and quality degrades. Here the low-precision form is what was trained, so the published weights are the intended artefact rather than a lossy convenience — which is also why ~1.4 TB is the real footprint rather than a starting point for further compression.
More on Kimi K3
Start with the complete Kimi K3 guide for the overview, or go deeper:
Ready to go deeper?
Read the full Kimi K3 guideFrequently Asked Questions
How many parameters does Kimi K3 have?
2.8 trillion in total, with 104 billion activated per token. It is a Mixture-of-Experts model with 896 experts, 16 of which are selected per token — making it the first open-weight model in the 3-trillion-parameter class.
What is Kimi Delta Attention?
A hybrid linear attention mechanism that replaces standard quadratic attention in most of K3's layers — 69 of its 93 layers use KDA, with 24 using Gated MLA and one dense. Linear attention scales much better with sequence length, which is what makes a 1M-token window practical rather than nominal.
Why is Kimi K3 cheap for its size?
Sparsity. Only 104B of 2.8T parameters activate per token, so compute per token resembles a much smaller model while total capacity stays large. Moonshot reports roughly 2.5x better scaling efficiency than Kimi K2. The trade-off is memory: all 2.8T parameters must be resident to serve any token.
