Kimi K3: Architecture, Training, Terminology and Local Deployment
Kimi K3 is Moonshot AI’s largest open-weight model to date: roughly 2.8 trillion total parameters, a Mixture-of-Experts design that activates about 104 billion parameters per token, native text and vision, and a maximum context of 1,048,576 tokens.
How to read this guide
| Part | Sections | Focus |
|---|---|---|
| A · Identity | 1–3 | Open-weight, decoder, multimodal, agentic; config; parameters |
| B · Architecture | 4–16 | KDA, MLA, NoPE, AttnRes, LatentMoE, routing, vision, optimiser |
| C · Training | 17–29 | Data, long context, SFT, RL, distillation, quantisation, infra |
| D · Deployment | 30–51 | Hardware, API, vLLM, SGLang, caching, APIs, troubleshooting |
| E · Strategy | 52–Conclusion | Why the design matters; references |
Introduction
Moonshot describes Kimi K3 as the first open model in the three-trillion-parameter class. Its architecture, training and deployment stack are oriented toward:
- long-running software-engineering tasks
- autonomous and semi-autonomous agents
- research and professional knowledge work
- multimodal reasoning over text, images and video
- terminal and application tool use
- million-token workflows
- complex reasoning with configurable thinking effort
Core architectural components:
- Kimi Delta Attention (KDA)
- Gated Multi-head Latent Attention (Gated MLA)
- Attention Residuals (AttnRes)
- Stable LatentMoE
- MoonViT-V2 for visual understanding
- SiTU-GLU activations
- Quantile Balancing for expert routing
- MXFP4 / MXFP8 quantisation for deployment
Scaling efficiency vs inference speed. “2.5× better scaling efficiency than Kimi K2” means converting training computation into model quality more efficiently. It does not mean every Kimi K3 response is 2.5× faster, or that inference costs are automatically 2.5× lower.
1. What kind of model is Kimi K3?
Kimi K3 can be described as:
Open-weight, decoder-based, natively multimodal, sparse Mixture-of-Experts reasoning and agentic foundation model.
Each part of that description has a specific meaning.
1.1 Open-weight
Open-weight means the trained numerical parameters are published and can be downloaded. That allows organisations to:
- self-host the model
- inspect its model configuration
- fine-tune or adapt it
- build specialised inference systems
- deploy it within private infrastructure
- develop derivative models, subject to the licence
Open-weight does not necessarily mean that:
- all training data is public
- the complete training code is available
- training can easily be reproduced
- deployment is inexpensive
- the licence is identical to Apache 2.0 or MIT
Kimi K3 uses a custom Kimi K3 License. Commercial users should review its conditions rather than assuming that “open” means unrestricted use.
1.2 Decoder-based model
A decoder model predicts the next token from the tokens that came before it.
This is autoregressive generation. Even when Kimi K3 produces a long answer, source code or a tool call, the output is generated token by token.
1.3 Native multimodality
Kimi K3 contains a visual encoder inside the model system rather than relying only on a separate image-captioning service. Its native visual pathway can process:
- text
- images
- scanned documents
- diagrams, charts, screenshots
- rendered webpages
- video frames
Moonshot’s hosted API supports images and uploaded videos. The current open-source serving contract documented by SGLang supports image input, but not yet video or audio in the self-hosted processor.
1.4 Agentic model
An agentic model is trained to work through a repeated action loop rather than produce only a single answer.
External tools might include a terminal, Python interpreter, browser, code editor, database, spreadsheet, MCP server, internal API or file system. The model does not automatically possess these tools — the agent framework exposes them and executes requested actions.
Moonshot trained Kimi K3 on trajectories involving reasoning, action, observation, verification and adaptation, sometimes across hundreds or thousands of tool calls and millions of accumulated context tokens.
2. Kimi K3 model configuration
| Component | Kimi K3 configuration |
|---|---|
| Architecture | Mixture-of-Experts |
| Total parameters | 2.8 trillion |
| Activated parameters | ~104 billion |
| Transformer layers | 93 |
| Dense layers | 1 |
| KDA layers | 69 |
| Gated MLA layers | 24 |
| Main hidden dimension | 7,168 |
| Attention heads | 96 |
| Latent MoE dimension | 3,584 |
| Expert hidden dimension | 3,072 |
| Routed experts | 896 |
| Routed experts selected per token | 16 |
| Shared experts | 2 |
| Vocabulary size | ~160,000 |
| Maximum context | 1,048,576 tokens |
| Visual encoder | MoonViT-V2 |
| Visual-encoder parameters | ~401 million |
| Visual-encoder layers | 27 |
| Visual patch size | 14 |
| Quantised expert weights | MXFP4 |
| Activations | MXFP8 where applicable |
| Primary modalities | Text and image |
These numbers describe different parts of the network and should not be treated as interchangeable.
3. Understanding parameters
3.1 What is a parameter?
A parameter is a learned numerical value inside a neural network. During training, the model adjusts parameters to reduce prediction error. Parameters encode statistical patterns from training data — word relationships, programming structures, maths, visual features, document layouts, tool-use behaviour and reasoning strategies.
A parameter is not equivalent to a stored fact. More parameters mean more representational capacity, but parameter count alone does not guarantee accuracy or intelligence.
3.2 Total parameters versus activated parameters
Kimi K3 contains about 2.8 trillion total parameters, but only about 104 billion are active while processing a particular token, because of sparse MoE layers.
Useful analogy — a large consulting organisation:
- The company employs 896 specialist teams.
- Every question does not go to all 896 teams.
- A router selects 16 relevant specialist teams.
- Two generalist teams participate in every case.
- Selected teams process the problem and return contributions.
All 896 experts contribute to total capacity; only a subset runs per token. The result is more stored capacity than a dense 104B model, without requiring all 2.8T parameters in every computation.
3.3 Does activating only 104B make the model “small”?
No. Even with sparse activation:
- the entire 2.8T checkpoint must still be stored
- expert parameters must be available across the GPU cluster
- the router must direct tokens to the relevant devices
- expert outputs must be transferred and combined
- attention states and caches still consume memory
- different tokens in the same batch may select different experts
Sparse activation reduces computation relative to a dense 2.8T model; it does not remove the model-storage requirement.
4. High-level architecture
Kimi K3 improves information flow across three dimensions:
| Dimension | Architectural component | Purpose |
|---|---|---|
| Sequence length | KDA and Gated MLA | Move information across short and extremely long token sequences |
| Network depth | Attention Residuals | Recover useful representations from earlier model blocks |
| Model width | Stable LatentMoE | Access a very large expert parameter space efficiently |
Images and videos first pass through MoonViT-V2. Visual representations are projected into the same embedding space as text before entering the backbone.
5. Tokenisation, vocabulary and embeddings
5.1 Vocabulary size
Kimi K3 has a vocabulary of about 160,000 token units. A token is not necessarily a complete word. It may be a common word, part of a word, punctuation, whitespace, a number, a keyword, an operator or a multilingual character sequence.
Example sentence:
The model processes information.
might tokenise roughly as:
"The" | " model" | " processes" | " information" | "."
Actual tokenisation depends on the model’s tokenizer. A larger vocabulary can represent common multilingual and programming patterns with fewer fragments, but it also increases embedding and output layer size.
5.2 Embeddings
A token ID is an integer. The embedding layer maps each ID to a vector of 7,168 dimensions:
5.3 Hidden dimension
The hidden dimension of 7,168 is the primary width of each token representation through the backbone. It does not mean the model stores 7,168 facts per token — each token is a vector of 7,168 learned features (semantic meaning, syntax, language, position, vision, code structure, tool-call format, intermediate reasoning). Dimensions generally lack simple one-to-one human interpretations.
6. Kimi Delta Attention
KDA is one of Kimi K3’s central architectural components.
6.1 The problem with conventional attention
In full self-attention, every token may compare itself with every earlier token. For a sequence of n tokens, pairwise relationships grow approximately as n².
Moving from 10,000 → 100,000 tokens does not create ten times as many relationships — a naïve quadratic calculation can create about 100× as many. At ~1M tokens, full attention in every layer creates substantial compute and KV-cache cost.
6.2 KDA’s recurrent state
Instead of preserving full key/value representations for every previous token in every KDA layer, KDA maintains a fixed-size recurrent state:
S_t = retained previous state + new information from token t
Where:
- S_t is the memory state after token t
- the previous state holds compressed information from earlier tokens
- a retention factor controls what is remembered
- a write-strength factor controls how much new information is added
The official formulation uses query, key and value vectors, a recurrent state, channel-wise retention α, and write-strength β. The model then reads from the updated state using the query.
6.3 Channel-wise forgetting
KDA does not apply a single forgetting rate to the whole hidden representation. It calculates retention per channel — one feature set can persist for thousands of tokens while another is replaced quickly.
Example during coding:
- project architecture may stay relevant for thousands of tokens
- the latest compiler error may matter only until fixed
- temporary terminal output may be discarded quickly
- user constraints may need persistent retention
The model learns these patterns rather than relying on hand-coded memory rules.
6.4 Delta-rule update
“Delta” means change. Instead of only appending, the delta-rule update modifies memory based on the difference between existing and incoming information — useful for updating an association rather than storing conflicting copies.
Existing memory:
Function X accepts two arguments.
New observation:
Function X was refactored and now accepts three arguments.
Delta update:
Modify the stored association for Function X.
The real state is numerical and distributed across channels, not readable sentences.
6.5 Short convolution
KDA applies a short convolution to query, key and value projections. A convolution looks at a small local neighbourhood — efficient access to local ordering before recurrent attention. Useful for adjacent words, programming syntax, maths, local visual-token sequences, punctuation and formatting.
6.6 L2 normalisation
Queries and keys are L2-normalised so vector length becomes one:
x̂ = x / ||x||₂
This prevents attention from being dominated by large-magnitude vectors and keeps similarity direction more consistent.
6.7 Chunkwise processing
Pure recurrence would process tokens sequentially (GPU-unfriendly). KDA uses a chunkwise formulation:
- tokens inside a chunk process in parallel
- recurrent state passes between chunks
- previous chunks enter via the incoming state
- interactions inside the current chunk are parallel
Moonshot developed specialised FlashKDA kernels to overlap chunk computation with recurrent-state propagation.
6.8 Benefits of KDA
- approximately linear long-sequence processing
- fixed-size recurrent state instead of ever-growing KV cache for KDA layers
- data-dependent forgetting
- support for extremely long contexts
- efficient prefix reuse
- position-sensitive behaviour without traditional positional encoding
6.9 Limitations of recurrent attention
A compressed recurrent state cannot preserve every exact token-to-token relationship as flexibly as global attention. Kimi K3 therefore periodically inserts Gated MLA layers for global interaction.
7. Gated Multi-head Latent Attention
7.1 What is attention?
Attention lets a token retrieve relevant information from other tokens. In:
Alice gave the document to Sarah because she requested it.
attention helps decide what “she” refers to.
7.2 Multi-head attention
Kimi K3 uses 96 attention heads. Each head can specialise (local syntax, entities, long-distance references, code dependencies, maths, layout, image-region relationships). Outputs are combined.
7.3 KV cache
During generation, attention layers reuse keys and values for earlier tokens — the key-value (KV) cache. Without it, the model would recalculate earlier representations every step. Cache memory grows with sequence length, layers, heads, head dimension and concurrent requests.
7.4 Multi-head Latent Attention
MLA compresses each token’s key/value representation into a smaller latent vector. Instead of storing full per-head K/V, MLA stores a compressed form and reconstructs when needed:
This reduces KV-cache memory while preserving global attention.
7.5 Gated output
Kimi K3 adds an input-dependent output gate to MLA. A sigmoid gate produces values in [0, 1]:
- near 0 — suppress the channel
- near 1 — allow the channel
- intermediate — partially preserve
Each token controls how much global-attention information it receives.
8. Why Kimi K3 combines KDA and MLA
Each main block roughly follows:
The model contains:
- 69 KDA layers
- 24 Gated MLA layers
- one final Gated MLA at the end of the backbone arrangement
- 93 attention layers in total
| Layer type | Main role |
|---|---|
| KDA | Efficiently carry and update long-running sequential state |
| Gated MLA | Re-examine global relationships among tokens |
| Stable LatentMoE | Specialised nonlinear feature processing |
The hybrid keeps much of full attention’s global reasoning capacity without quadratic attention in every layer.
9. No Position Encoding (NoPE)
Many transformers use positional encodings such as RoPE. Kimi K3 uses NoPE — no explicit conventional positional encoding in MLA layers. Position and recency come mainly from KDA’s recurrent processing, retention gates, decay and short convolutions.
That makes progressive context extension less dependent on retuning positional-encoding parameters. RoPE models often need frequency-base adjustment, interpolation or YaRN when extending context. Kimi K3 avoids that particular requirement, though it still had to be trained on progressively longer contexts.
10. Attention Residuals
10.1 Conventional residual connections
Transformer layers usually use residuals:
x_{l+1} = x_l + F_l(x_l)
Where x_l enters layer l, and F_l is the layer transform. Residuals help gradients flow, preserve earlier information, stabilise training and support depth.
10.2 The dilution problem
With fixed-weight accumulation over many layers, hidden-state magnitude can grow, earlier contributions dilute, later layers only see an accumulated mixture, and a later layer cannot explicitly choose a particular earlier representation. With 93 layers, depth-wise information flow matters.
10.3 How Attention Residuals work
AttnRes replaces fixed accumulation with learned attention over previous representations. A later layer may selectively retrieve from the original embedding, the current block, earlier blocks, recent or distant representations — using learned pseudo-queries.
Instead of “use only the latest accumulated representation,” the model can blend sources with learned, input-dependent weights (for example, more weight on an earlier block, some on the embedding, some on recent state).
10.4 Block Attention Residuals
Attending to every earlier layer would cost large memory and inter-device communication. Kimi K3 uses block-level AttnRes: layers are grouped; later modules attend over block representations. Most of the selective-retrieval benefit remains with lower overhead.
10.5 Practical importance
AttnRes can help recover exact low-level token details, earlier visual information, intermediate reasoning, diluted representations, original user constraints and structural code information.
- AttnRes → information flow through depth
- KDA → information flow through sequence length
11. Stable LatentMoE
Stable LatentMoE is Kimi K3’s feed-forward expert architecture.
11.1 Feed-forward networks
After attention mixes information among tokens, transformers usually apply feed-forward networks.
- Attention answers: Which information from other tokens is relevant?
- The feed-forward network answers: How should this token’s features be transformed?
In Kimi K3, most feed-forward computation is provided by experts.
11.2 Routed experts
Kimi K3 contains 896 routed experts. A router assigns each token to 16 of them. Selected experts process the token independently; weighted outputs are combined:
Specialisation emerges statistically during training. Experts are not necessarily labelled as explicit human categories such as “Python expert” or “finance expert.”
11.3 Shared experts
Two shared experts process every token. Shared experts can learn broadly useful capabilities; routed experts handle more specialised transforms. Final result:
Shared-expert output + Routed-expert output
11.4 Latent MoE dimension
The main token representation has 7,168 dimensions. Instead of sending the full width to every routed expert, Kimi K3 projects into a 3,584-dimensional latent space:
This reduces communication and processing cost on the routed path. The two shared experts still operate on the full-width representation.
11.5 Why 16 experts instead of one or two?
Activating more experts lets the model combine multiple specialised transforms for a token (for example syntax, repository context, security, dependencies and tool planning at once). More experts also increase computation, routing complexity, inter-GPU communication and load-balancing difficulty. The latent bottleneck makes activating 16 experts more practical.
12. Expert routing and Quantile Balancing
12.1 The load-balancing problem
Without balancing, a router may repeatedly select a few popular experts. That overloads some GPUs, underuses others, over-trains popular experts, under-trains unpopular ones, slows distributed training and harms specialisation.
12.2 Quantile Balancing
Kimi K3 uses Quantile Balancing to adjust expert-routing biases. Target assignments per expert depend on batch tokens (m), experts selected per token (k) and total experts (n):
q = (m × k) / n
Quantile Balancing estimates the score threshold that would give each expert roughly its target load. Router bias updates for the next training step; the final routing bias is frozen during inference.
12.3 Simple analogy
Imagine 896 service teams. If Team 14 gets too many tickets, raise its assignment threshold. If Team 562 gets too few, lower its threshold. The goal is useful specialisation without workload collapse onto a few experts — not random assignment.
13. RMSNorm
RMSNorm (Root Mean Square Normalisation) controls representation scale:
RMSNorm(x) = x / sqrt(mean(x²) + ε)
Where ε is a small stabilising constant. Kimi K3 inserts RMSNorm between routed-expert aggregation and the up-projection. That reduces sensitivity to different selected experts, routing weights, large expert outputs and changing token distributions. Moonshot reports improved training stability, validation loss and downstream performance.
14. SiTU-GLU
14.1 Gated Linear Units
A GLU has a value branch and a gating branch:
output = value(x) ⊙ gate(x)
The gate decides how much of each value channel should pass.
14.2 Why replace SwiGLU?
Many language models use SwiGLU. It performs well but is unbounded — large inputs can produce very large outputs. At 2.8T scale, activation growth can contribute to numerical instability.
14.3 Sigmoid Tanh Unit GLU
Kimi K3 uses SiTU-GLU (Sigmoid Tanh Unit GLU). It is designed to:
- behave similarly to SwiGLU around normal inputs
- remain bounded for extreme values
- reduce activation explosion
- improve stability under heavy quantisation
- support very sparse MoE training
The technical report describes SiTU-GLU as bounded at large positive inputs, unlike unbounded SwiGLU.
15. MoonViT-V2 visual encoder
15.1 Vision Transformer
MoonViT-V2 is a Vision Transformer. It divides an image into patches, converts each patch to a vector, and processes patch vectors with transformer layers.
Kimi K3 uses patch size 14. The visual encoder has about 401M parameters, 27 layers and 12 visual-attention heads.
15.2 Projector
Visual-encoder output is not automatically in text embedding space. A lightweight projector maps visual features into the shared embedding space so the backbone can reason jointly over text tokens, image tokens, layout features and video-frame representations.
15.3 Pixel shuffle
Kimi K3 uses 2 × 2 pixel-shuffle downsampling before projection, cutting visual tokens by about 4×. High-resolution images would otherwise consume a large fraction of context and attention cost. The report describes inputs up to about 3,584 × 3,584 pixels within relevant processing constraints.
15.4 Video processing
For video:
- spatial attention processes relationships within frames
- temporal attention processes relationships across frames
- temporal pooling reduces frame-token count
The same broad visual encoder serves image and video. The hosted Moonshot API supports video-file upload; current self-hosted open-source documentation says the open processor supports image input only.
15.5 Native multimodal training
Moonshot trained MoonViT-V2 from scratch with the next-token-prediction objective rather than relying only on a separately contrastively trained vision model. Moonshot reports more stable visual-encoder gradient behaviour while matching its SigLIP-initialised comparison across vision evaluations.
16. Per-Head Muon optimiser
An optimiser determines how parameters update during training. Kimi K3 uses Muon for matrix parameters and a per-attention-head variation for query, key and value projection matrices.
16.1 Momentum
Optimisers often keep a moving estimate of past gradients (momentum) to smooth noisy updates, accelerate useful directions and reduce oscillation.
16.2 Orthogonalisation
Muon applies Newton–Schulz orthogonalisation to update matrices — controlling the geometry of matrix updates rather than applying raw gradients without structural normalisation.
16.3 Per-head processing
Instead of orthogonalising the full attention-projection matrix as one unit, Kimi K3 divides by attention head. That prevents heads with larger gradient scales from dominating smaller ones. Moonshot reports more balanced learning across heads, better large-scale stability and slightly lower optimiser overhead.
17. How Kimi K3 was built
18. Architecture and scaling-law research
Moonshot did not simply increase Kimi K2’s parameter count. It evaluated depth, expert count, selected experts, active parameters, attention heads, latent expert dimensions, learning-rate schedules, batch sizes, tokens-per-parameter ratios, model shape and data recipes.
Scaling laws estimate how validation loss changes with model size, training tokens, computation and architecture. Moonshot compared K2 and K3 curves and reported that K3 reached equivalent validation-loss levels with about 2.5× greater scaling efficiency.
Interpret as:
Under Moonshot’s scaling-law experiments, Kimi K3’s design converted pre-training computation into model quality more effectively than Kimi K2’s design.
It does not imply that inference is always 2.5× faster, API usage is 2.5× cheaper, every benchmark is 2.5× higher, or the model needs 2.5× fewer GPUs.
19. Pre-training data
Text pre-training covered four broad domains:
- web text
- programming code
- mathematics
- knowledge-oriented content
Visual corpus included image captions, interleaved image-text documents, OCR, visual-perception data, videos, rendered webpages, SVG, games, 3D assets, visual coding examples and CAD schematics.
Moonshot combined heuristic filtering, classifier-based quality scoring, exact and fuzzy deduplication, domain-specific sampling, synthetic-data generation and source-fidelity checks.
19.1 Deduplication
Removes repeated or near-repeated examples. Without it, models may memorise repeats, waste compute, contaminate benchmarks and lose diversity.
19.2 Quality classifiers
Estimate whether a document is suitable for training — coherence, formatting, completeness, language quality, information density, spam probability, code correctness, and similar signals.
19.3 Rephrased data
Moonshot describes rephrasing knowledge and maths content in multiple styles and perspectives, then verifying fidelity against original sources — more variety without changing underlying facts.
19.4 Programmatic multimodal data
Training paired code with rendered outputs:
That helps connect implementation choices to visual consequences — important for vision-in-the-loop coding agents.
20. Native multimodal pre-training
Language and visual pathways were jointly optimised during large-scale training — unlike a simpler path where a language model is trained, a vision encoder is trained separately, the two are connected later, and only a small projector is adapted.
Native multimodal training helps the backbone learn relationships such as code → rendered UI, chart → textual interpretation, document image → structure, video frames → temporal events, screenshot → application state, and visual error → code correction.
21. Progressive long-context training
Kimi K3 was not trained at one million tokens from step one. Moonshot used a progressive curriculum:
Shorter sequences came first; longer contexts arrived later, especially in cooldown. Advantages: cheaper early stages, general patterns first, expensive million-token training concentrated later, gradual context growth, more stable optimisation.
Moonshot states context was extended from 8K → 64K during pre-training, then 256K → 1M during cooldown.
21.1 Why long documents alone are insufficient
A one-million-token document does not automatically teach million-token reasoning — a model could focus only on nearby text. Moonshot created synthetic long-context tasks with required information distributed across the sequence:
Critical instruction at token 20,000
Relevant table at token 420,000
Exception at token 790,000
Question at token 995,000
The task requires integrating distant sections.
21.2 Long-data cleaning
Very long documents and videos may contain duplicates, binary content, truncated data, corrupt logs, low-quality video segments and repetitive machine-generated material. Moonshot used dedicated long-context cleaning, structural validation and perceptual hashing for video frames.
22. Supervised fine-tuning
After pre-training, Kimi K3 underwent supervised fine-tuning (SFT).
22.1 What SFT does
Pre-training teaches general next-token prediction. SFT teaches how to respond to instructions in an intended format:
22.2 Agentic SFT data
Moonshot expanded SFT for complex agentic tasks. Trajectories from specialised earlier Kimi models went through automated verification, multiple verification stages, human-in-the-loop annotation and consistent conversation serialisation — establishing adaptive reasoning, precise tool calling, long-horizon execution and structured agent behaviour.
22.3 Cold-start policy
The SFT model is a cold-start policy for reinforcement learning: a competent initial model. Without it, RL would need to rediscover basics such as following instructions, tool syntax, valid code, reading tool output and ending tasks properly.
23. Reinforcement learning
RL trains using rewards on actions and outcomes:
Kimi K3’s RL covered three broad areas.
23.1 General tasks
General reasoning, knowledge, visual reasoning, faithfulness, search and professional knowledge work.
23.2 General agents
Long-horizon assistant tasks, deep research, professional workflows, extended writing and tool-use sequences.
23.3 Coding agents
Software engineering, coding, GPU-kernel tasks, web development, compiler and system tasks, and vision-in-the-loop programming.
Each domain was trained at three reasoning-effort levels — low, high, max — creating nine specialist RL policies:
3 domains × 3 effort levels = 9 policies
Moonshot reports that increasing RL computation produced higher capability scores and longer successful tool-use trajectories.
24. Reasoning-effort training
Kimi K3 supports low, high and max. Reasoning effort controls how much test-time computation the model allocates.
24.1 Token budgets during RL
Moonshot assigned each problem an estimated token budget. Trajectories that exceeded a scaled budget threshold could receive a negative reward. Training started with a larger maximum reasoning budget, then progressively smaller budgets for high- and low-effort policies — teaching problem-solving under different compute constraints.
24.2 Practical interpretation
| Reasoning effort | Appropriate use |
|---|---|
| Low | Extraction, classification, simple rewriting, straightforward questions |
| High | Coding, document analysis, research, multi-step planning |
| Max | Difficult debugging, complex reasoning, high-value long-horizon tasks |
Higher effort may increase output tokens, latency, cost and tool interactions. It does not guarantee every answer is correct.
25. Multi-Teacher On-Policy Distillation (MOPD)
Nine specialised RL policies could be separate models, but that is operationally complex. Moonshot consolidated them into one Kimi K3 using Multi-Teacher On-Policy Distillation (MOPD).
25.1 Distillation
Transfers behaviour from teacher model(s) into a student — matching output probabilities, responses or reasoning trajectories.
25.2 Multiple teachers
Teachers covered combinations of task domain and reasoning effort. The unified model learned from their trajectories so one deployment supports all three effort levels and multiple domains.
25.3 On-policy
Training data is generated in a manner aligned with the current policy, not only old static datasets — reducing mismatch between training situations and the situations the model creates through its own behaviour.
26. Quantisation-aware post-training
26.1 Why quantisation was required
A 2.8T model at 16 bits per parameter would need several terabytes of raw weight memory. MoE expert weights dominate. Moonshot quantised those expert weights with MXFP4.
26.2 MXFP4
MXFP4 is a four-bit microscaling floating-point format. Four-bit storage substantially reduces memory, storage, bandwidth and expert-weight transfer cost. “Microscaling” means small groups share scaling information so low-bit values still cover useful numerical ranges.
26.3 MXFP8 activations
Activations are temporary values during processing. Kimi K3 uses MXFP8 activations in relevant expert computation — more range/precision than 4-bit, more efficient than BF16/FP16.
26.4 Higher-precision components
Not everything is 4-bit. Higher precision remains for components such as attention projections, latent MoE projections, shared experts and MoE routers. Quantisation primarily targets routed expert weights that dominate parameter memory.
26.5 Quantisation-aware training
Kimi K3 did not only train at high precision and compress afterward. Quantisation-aware training ran through SFT and RL so parameters adapt to rounding, reduced precision, quantisation noise and limited ranges. The same quantisation arrangement was used during RL rollout and training to avoid train–inference mismatch.
27. Multi-token prediction and speculative decoding
Kimi K3 was pre-trained with a multi-token-prediction layer. Ordinary generation predicts one next token at a time. A draft model can propose several future tokens:
This is speculative decoding. Kimi K3’s multi-token-prediction layer was adapted into an EAGLE-3-style draft model — improving generation speed without changing the final target distribution when verification is correct.
Speculative decoding helps most when draft-token acceptance is high, batches are not extremely large, prompts are not overwhelmingly long, and decode time dominates. Benefit may decline for long-context, prefill-heavy requests.
28. Training infrastructure
Training a 2.8T multimodal MoE requires distributed parallelism. Moonshot combined pipeline parallelism, virtual pipeline stages, expert parallelism, data parallelism, ZeRO sharding, context parallelism and KDA context parallelism.
28.1 Data parallelism
Multiple model replicas process different examples; gradients are combined. Increases throughput but needs memory for replicas or sharded states.
28.2 Tensor parallelism
Divides individual matrix operations across GPUs (for example by columns or rows). Useful when a single layer cannot fit or run efficiently on one GPU.
28.3 Pipeline parallelism
Divides layers into stages:
Microbatches flow through the pipeline. Reduces layers per GPU; can create idle pipeline bubbles.
28.4 Virtual pipeline stages
Divide a physical stage into smaller logical sections for more interleaving and less idle time.
28.5 Expert parallelism
Distributes MoE experts across GPUs. Tokens go to GPUs that hold selected experts; results return and combine — often via all-to-all communication.
28.6 All-to-all communication
Every participating GPU may send data to every other participating GPU — dispatching tokens to expert hosts and returning outputs. Poor all-to-all performance is a major MoE bottleneck.
28.7 Context parallelism
Divides a long input sequence across devices. Different GPUs process different sequence sections and exchange information needed for correctness.
28.8 KDA context parallelism
KDA’s recurrent state makes splitting sequences harder than splitting independent tokens. Moonshot’s KDA Context Parallelism represents sequence segments as composable state transformations combined with a prefix-scan-style operation — scaling long-sequence processing while exchanging fixed-size state fragments.
28.9 ZeRO
ZeRO reduces memory duplication across data-parallel workers by sharding optimiser states, gradients and/or parameters. Moonshot reports ZeRO-1 data parallelism and pipeline ZeRO-2 gradient sharding as part of the Kimi K3 training system.
29. Long-horizon RL infrastructure
Agentic RL differs from standard Q&A training. A trajectory may include hundreds of terminal commands, file reads, web interactions, screenshots, tool outputs, millions of cumulative tokens and a persistent software environment.
29.1 Partial rollouts
Completed parts of long trajectories can train while unfinished trajectories continue across later iterations — reducing the impact of extremely long tasks delaying an entire batch.
29.2 External KV-cache pool
At million-token scale, recalculating an earlier prefix is expensive. Moonshot moved reusable but inactive prefixes from GPU to CPU memory and restored them when needed (write-back design):
- active cache blocks stay on GPU
- evicted reusable prefixes move to CPU DRAM
- prefixes are prefetched before reuse
- KDA states and MLA KV blocks are managed together
29.3 Resumable microVM sandboxes
Coding and agentic tasks run in isolated environments (repository, dependencies, files, terminal history, services, partial state). Resumable microVM sandboxes let long tasks continue without rebuilding from scratch. The report states more than 51 million sandbox instances across more than 1.5 million images were created during training and evaluation.
30. Can Kimi K3 run locally?
The answer depends on what “locally” means.
30.1 Local application using the hosted API
Run your app on your laptop or server while sending inference to Moonshot’s hosted API:
Requirements: Python 3.9+, API key, internet, OpenAI Python SDK. The model itself does not run on your hardware.
30.2 True local or self-hosted deployment
True local deployment means downloading the complete model, loading weights into a GPU cluster, running vLLM / SGLang / another supported engine, exposing your own endpoint, and managing memory, caching, networking and scaling.
This cannot realistically be done on an ordinary laptop, MacBook, single RTX 4090, normal desktop, or a conventional one- or two-GPU server.
The current vLLM recipe lists an approximately 1,680 GB MXFP4 model variant and recommends at least eight GB300-class GPUs, with multi-node deployment for production traffic.
31. Officially documented hardware shapes
Current SGLang deployment cookbook topologies:
| Hardware | Documented topology | Total GPUs |
|---|---|---|
| NVIDIA B300 | 1 node × 8 GPUs | 8 |
| NVIDIA GB300 | 2 nodes × 4 GPUs | 8 |
| NVIDIA B200 | 2 nodes × 8 GPUs | 16 |
| NVIDIA GB200 | 4 nodes × 4 GPUs | 16 |
| NVIDIA H200 | 2 nodes × 8 GPUs | 16 |
| NVIDIA H100 | 4 nodes × 8 GPUs | 32 |
| AMD MI350X/MI355X | 1 node × 8 GPUs | 8 |
These are starting points, not universal guarantees. SGLang notes some serving rounds and presets remain under final verification — re-measure for your workload.
31.1 Why H100 needs more GPUs
An H100 commonly provides 80 GB HBM per GPU. A 32-GPU cluster provides about:
32 × 80 GB = 2,560 GB
That must hold ~1,680 GB of checkpoint data plus runtime representations, quantisation metadata, communication buffers, KDA states, MLA KV cache, visual encoder, CUDA graphs/kernels and active-request memory. Raw model size is not the full GPU-memory requirement.
31.2 Why eight B300 or MI355X GPUs can be sufficient
Newer high-memory accelerators provide more HBM per GPU. For example:
8 × 268 GB = 2,144 GB
8 × 288 GB = 2,304 GB
Usable memory is lower than nominal totals; practical concurrency depends on context, cache precision and engine settings.
32. Recommended self-hosting configuration
32.1 GPU
Minimum practical documented options include:
- 8 × NVIDIA B300
- 8 × NVIDIA GB300 across two systems
- 8 × AMD MI350X or MI355X
- 16 × NVIDIA H200
- 16 × NVIDIA B200
- 32 × NVIDIA H100
32.2 Storage
The current vLLM recipe lists the MXFP4 variant at ~1,680 GB. Plan extra space for partial downloads, Hugging Face cache, container images, compiled kernels, temp files, logs and alternate checkpoints.
An engineering allowance of about 2.5–3 TB of fast NVMe is reasonable for one model copy plus headroom — a capacity-planning recommendation, not an official Moonshot minimum.
32.3 Host memory
System RAM serves model-loading buffers, CPU-side cache, tokenisation, image processing, host KV-cache offload, serving processes and OS caching. There is no single universal RAM figure; large deployments should expect hundreds of GB per node (more with substantial CPU-side prefix caches).
32.4 Network
For multi-node configs use high-bandwidth, low-latency interconnect: InfiniBand, RDMA-capable Ethernet, NVLink/NVSwitch within nodes, MNNVL on supported Grace Blackwell systems. The vLLM recipe states its multi-node tensor-parallel setup requires InfiniBand. Ordinary 1 Gb or 10 Gb Ethernet is unlikely to be enough for production expert-routing and tensor-parallel traffic.
32.5 Operating system
Use a supported Linux distribution with current GPU drivers, CUDA or ROCm, Docker, NVIDIA Container Toolkit (for NVIDIA), sufficient shared memory and Hugging Face credentials where required.
32.6 Serving engine
Moonshot currently recommends vLLM, SGLang and TokenSpeed. Model cards and deployment docs provide current Kimi-specific recipes.
33. Running Kimi K3 through the hosted API
Easiest path to use Kimi K3.
Step 1 — Install Python and the SDK
python3 -m pip install --upgrade "openai>=1.0"
Step 2 — Configure the API key
On Linux or macOS:
export MOONSHOT_API_KEY="your-api-key"
Do not hard-code production API keys into source code.
Step 3 — Create a client
from __future__ import annotations
import os
from openai import OpenAI
api_key = os.environ.get("MOONSHOT_API_KEY")
if not api_key:
raise RuntimeError("MOONSHOT_API_KEY is not configured.")
client = OpenAI(
api_key=api_key,
base_url="https://api.moonshot.ai/v1",
timeout=3600,
)
Step 4 — Send a request
response = client.chat.completions.create(
model="kimi-k3",
reasoning_effort="high",
messages=[
{
"role": "user",
"content": (
"Explain the difference between tensor parallelism "
"and expert parallelism."
),
}
],
)
message = response.choices[0].message
print("Final answer:")
print(message.content)
Moonshot requires Python 3.9+ in quickstart examples. Kimi K3 always operates in thinking mode and accepts low, high and max reasoning effort, with max as the default.
34. Preserving thinking history
Kimi K3 returns separate fields for reasoning_content, content, and tool_calls where applicable.
In multi-turn conversations, Moonshot requires the complete assistant message to be passed back. Do not preserve only assistant_message.content — preserve the complete message object.
from typing import Any
messages: list[Any] = [
{
"role": "user",
"content": "Review this architecture and identify three risks.",
}
]
first = client.chat.completions.create(
model="kimi-k3",
reasoning_effort="high",
messages=messages,
)
assistant_message = first.choices[0].message
messages.append(assistant_message)
messages.append(
{
"role": "user",
"content": "Expand the second risk into a mitigation plan.",
}
)
second = client.chat.completions.create(
model="kimi-k3",
reasoning_effort="high",
messages=messages,
)
print(second.choices[0].message.content)
This preserves the reasoning state expected by the multi-turn training format.
35. Self-hosting with vLLM
The Kimi-specific vLLM container is the safest starting point — recent support with model-specific kernels and integration work.
35.1 Example single-node starting template
Starting template for an eight-GPU high-memory system. Validate against the current vLLM deployment page for your hardware.
export HF_TOKEN="your-hugging-face-token"
docker run --rm \
--gpus all \
--privileged \
--ipc=host \
-p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_TOKEN="$HF_TOKEN" \
-e VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 \
vllm/vllm-openai:kimi-k3 \
moonshotai/Kimi-K3 \
--trust-remote-code \
--load-format fastsafetensors \
--gpu-memory-utilization 0.95 \
--tensor-parallel-size 8 \
--enable-auto-tool-choice \
--tool-call-parser kimi_k3 \
--reasoning-parser kimi_k3
The recipe recommends the Kimi-specific image, model-specific reasoning/tool parsers, and the latent-MoE tail-fusion environment variable. It also warns that Kimi K3 may occasionally produce tool-call output that does not match the parser — applications should validate schemas and retry safely.
35.2 What the command means
| Flag / setting | Meaning |
|---|---|
--gpus all | All host GPUs visible to the container |
--privileged | Extra device/system access for some multi-GPU ops — controlled environments only |
--ipc=host | Share host IPC namespace (shared memory for GPU workers) |
-p 8000:8000 | Endpoint at http://localhost:8000/v1 |
| HF cache mount | Persist downloads on the host |
HF_TOKEN | Hugging Face credential — treat as a secret |
VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 | Fused LatentMoE tail path (fewer intermediate memory trips) |
--trust-remote-code | Load custom modelling code — only for trusted repos/revisions |
--load-format fastsafetensors | Fast safetensors load path |
--gpu-memory-utilization 0.95 | Up to ~95% GPU memory for model + caches |
--tensor-parallel-size 8 | Split tensor ops across eight GPUs |
--enable-auto-tool-choice | Model may issue tool calls when tools are supplied |
--tool-call-parser kimi_k3 | Map generated tool calls to OpenAI-compatible tool_calls |
--reasoning-parser kimi_k3 | Separate reasoning from final answer |
36. Multi-node vLLM deployment
A documented example uses two nodes with eight GPUs each and tensor-parallel size 16. Simplified head-node shape:
export HEAD_IP="10.0.0.10"
export IFACE_NAME="ib0"
export HF_TOKEN="your-hugging-face-token"
docker run --rm \
--gpus all \
--privileged \
--ipc=host \
--network=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_TOKEN="$HF_TOKEN" \
-e GLOO_SOCKET_IFNAME="$IFACE_NAME" \
-e NCCL_SOCKET_IFNAME="$IFACE_NAME" \
-e VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 \
vllm/vllm-openai:kimi-k3 \
moonshotai/Kimi-K3 \
--trust-remote-code \
--load-format fastsafetensors \
--gpu-memory-utilization 0.95 \
--tensor-parallel-size 16 \
--nnodes 2 \
--node-rank 0 \
--master-addr "$HEAD_IP" \
--enable-auto-tool-choice \
--tool-call-parser kimi_k3 \
--reasoning-parser kimi_k3
On the second node, use the same cluster address with:
--node-rank 1
--headless
The recipe uses NCCL/Gloo interface variables and requires a high-performance interconnect for cross-node tensor parallelism. Exact kernel and MoE backend flags differ by GPU generation — treat the current generated recipe as source of truth.
36.1 NCCL
NVIDIA’s communication library for multi-GPU collectives: all-reduce, all-gather, reduce-scatter, all-to-all.
36.2 Gloo
Another distributed backend, often used for coordination and CPU-side distributed operations.
36.3 NCCL_SOCKET_IFNAME
Selects the NIC NCCL should use (for example ib0). Wrong interface may send traffic over ordinary Ethernet instead of InfiniBand.
36.4 Node rank
Each node gets a unique rank (0 = head, 1 = worker, …).
36.5 Master address
IP used to initialise the distributed process group. All nodes must reach it.
37. Self-hosting with SGLang
SGLang provides Kimi-specific Docker images for CUDA and ROCm. Current image tags include:
docker pull lmsysorg/sglang:kimi-k3
docker pull lmsysorg/sglang:kimi-k3-cu12
docker pull lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727
First is CUDA 13-oriented; second is CUDA 12; third targets supported AMD MI35x hardware.
37.1 Recommended high-speed B300 profile
sglang serve \
--trust-remote-code \
--model-path moonshotai/Kimi-K3 \
--tp-size 8 \
--dcp-size 8 \
--mem-fraction-static 0.85 \
--mm-feature-transport cuda_ipc \
--mm-processor-worker-num 2 \
--mm-io-worker-num 16 \
--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--host 0.0.0.0 \
--port 30000
Oriented toward high-speed multimodal serving on a single B300 node.
37.2 Important SGLang flags
| Flag | Meaning |
|---|---|
--tp-size 8 | Tensor parallelism across eight GPUs |
--dcp-size 8 | Decode-context parallelism — shards MLA KV cache across ranks |
--mem-fraction-static 0.85 | ~85% GPU memory for static model/cache structures |
--mm-feature-transport cuda_ipc | Multimodal features via CUDA IPC (single node) |
--mm-processor-worker-num 2 | Two multimodal preprocessing workers |
--mm-io-worker-num 16 | Sixteen I/O workers for image loading |
38. Lower-HBM SGLang profile
When memory headroom matters more than maximum concurrency:
SGLANG_VIT_ENABLE_CUDA_GRAPH=0 \
sglang serve \
--trust-remote-code \
--model-path moonshotai/Kimi-K3 \
--tp-size 8 \
--context-length 65536 \
--enable-symm-mem \
--mem-fraction-static 0.82 \
--mm-feature-transport cpu \
--mm-processor-worker-num 2 \
--mm-io-worker-num 16 \
--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--host 0.0.0.0 \
--port 30000
This reduces pressure by limiting context to 65,536 tokens, using CPU transport for visual features, lowering static-memory fraction and disabling vision-transformer CUDA graph capture. Documented as a conservative B300 starting point — not portable to every GPU.
39. Calling the locally hosted server
Both vLLM and SGLang expose an OpenAI-compatible endpoint.
vLLM
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
timeout=3600,
)
response = client.chat.completions.create(
model="moonshotai/Kimi-K3",
reasoning_effort="high",
messages=[
{
"role": "user",
"content": "Explain Kimi Delta Attention in simple terms.",
}
],
max_tokens=2048,
)
message = response.choices[0].message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)
SGLang
Change the endpoint to:
base_url="http://localhost:30000/v1"
SGLang’s Kimi K3 reasoning parser exposes reasoning separately from the final answer.
40. Sending an image to a local deployment
from __future__ import annotations
import base64
from pathlib import Path
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600,
)
image_path = Path("architecture.png")
if not image_path.exists():
raise FileNotFoundError(image_path)
encoded_image = base64.b64encode(image_path.read_bytes()).decode("utf-8")
response = client.chat.completions.create(
model="moonshotai/Kimi-K3",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{encoded_image}"
},
},
{
"type": "text",
"text": (
"Analyse this architecture diagram and explain "
"the data flow."
),
},
],
}
],
max_tokens=4096,
)
print(response.choices[0].message.content)
Moonshot’s hosted API requires base64 or its uploaded-file scheme rather than public image URLs. Self-hosted engines may accept public URLs depending on implementation and security configuration.
41. Context length versus memory allocation
The model supports a maximum of 1,048,576 tokens, but serving at the full limit has significant consequences.
41.1 Context length
Context includes system instructions, conversation messages, reasoning history, tool definitions, tool outputs, images-as-tokens and generated output. Input and output must fit the server’s configured capacity.
41.2 Maximum model length
Options such as --max-model-len or --context-length set the longest accepted request. Reducing them can shrink context buffers, simplify capacity planning, prevent extremely expensive requests and improve concurrency.
SGLang notes that context length alone does not directly size every cache pool — Kimi K3 has separate KDA-state and MLA-KV memory pools.
41.3 Start below one million tokens
For an initial deployment, begin with 32K, 64K or 128K. Benchmark time to first token, output-token speed, max concurrency, GPU-memory utilisation, cache-hit rate and response quality. Increase toward one million only when the workload genuinely needs it.
42. Prefill and decode
42.1 Prefill
Processes the existing prompt. For a 400,000-token input, the server must process those tokens before the first generated token (unless a prefix is cached). Prefill is usually compute-intensive, sensitive to prompt length, highly parallelisable and responsible for time to first token.
42.2 Decode
Generates new tokens one at a time. Usually memory-bandwidth intensive, latency sensitive, repeated per output token and responsible for inter-token latency.
42.3 Prefill/decode disaggregation
Large production systems may use separate GPU pools:
Each pool can be optimised for a different workload. Because Kimi K3 is hybrid, both MLA KV cache and KDA recurrent state must be transferred.
43. KDA state pool and MLA KV pool
43.1 KDA state pool
KDA layers maintain fixed-size recurrent states. The KDA pool determines how many active requests can hold required states simultaneously — an important concurrency ceiling in SGLang.
43.2 MLA KV pool
MLA layers maintain compressed per-token KV representations. This pool grows with sequence length, concurrent requests and KV precision.
43.3 Memory-ratio configuration
SGLang uses --mamba-full-memory-ratio to divide memory between the recurrent state pool and full-attention KV pool. Although KDA is not literally Mamba, serving frameworks often reuse hybrid-state infrastructure built for Mamba-like layers.
The right ratio depends on average request length, output length, speculative decoding, context parallelism, cache strategy, state precision and KV precision.
44. Prefix caching
Many agentic conversations repeatedly send a large unchanged prefix (system prompt, repo summary, tools, prior conversation, code context) while only the latest user message changes.
Prefix caching stores the processed representation of the unchanged prefix. A cache hit avoids repeating much of the prefill calculation.
Moonshot’s hosted API performs automatic prefix caching for prompts longer than 256 tokens when the prefix is unchanged.
Self-hosted engines must preserve both KDA recurrent state and MLA KV-cache blocks. A prefix is reusable only when both attention mechanisms can be restored consistently.
45. Hierarchical caching
Large deployments can store cache data across tiers:
| Tier | Storage | Characteristics |
|---|---|---|
| L1 | GPU HBM | Fastest, smallest, most expensive |
| L2 | Host RAM | Larger but slower |
| L3 | Distributed or NVMe-backed cache | Largest, highest latency |
A hierarchical system may keep active requests on GPU, move inactive prefixes to host memory, move colder prefixes to distributed storage, then restore them when the conversation continues.
SGLang’s HiCache and systems such as LMCache provide hybrid-model mechanisms; exact support varies by engine version and parallelism configuration.
46. Tensor, expert, data and decode-context parallelism
| Parallelism | What is divided? | Main purpose |
|---|---|---|
| Tensor parallelism | Matrix calculations | Fit and run large layers across GPUs |
| Pipeline parallelism | Model layers | Split network depth across stages |
| Expert parallelism | MoE experts | Distribute expert weights and computation |
| Data parallelism | Requests or training batches | Throughput via replicas |
| Context parallelism | Input sequence | Very long contexts across devices |
| Decode-context parallelism | MLA KV cache during decode | Reduce duplicated KV memory |
Production may combine several forms, for example:
Tensor parallelism: 16
Expert parallelism: 16
Data parallelism: 4
Decode-context parallelism: 8
Choose based on priority: lowest latency, highest throughput, longest context, highest concurrency or lowest serving cost.
47. Important API terminology
base_url
Location of the API server.
| Environment | Example |
|---|---|
| Hosted | https://api.moonshot.ai/v1 |
| Local vLLM | http://localhost:8000/v1 |
| Local SGLang | http://localhost:30000/v1 |
messages
Conversation history. Each message usually has role and content.
Roles
| Role | Purpose |
|---|---|
system | Persistent instructions or application policy |
user | User request |
assistant | Model’s previous response |
tool | Result of a tool call |
Other fields
| Field | Meaning |
|---|---|
reasoning_effort | Approximate reasoning budget: low, high, max |
reasoning_content | Model’s reasoning output |
content | Final user-facing answer |
stream=True | Incremental output instead of waiting for the full response |
max_tokens | Output limit on many OpenAI-compatible self-hosted servers |
max_completion_tokens | Hosted API: default 131,072; max 1,048,576 (request must still fit context/account limits) |
response_format | Constrains final response (for example strict JSON schema) |
tool_choice | Tool-use behaviour: auto, required, none, or a named function |
tool_call_id | Connects a tool result to the original tool request |
Every returned tool call should receive one matching tool response.
48. Structured output
Structured output forces the final answer to follow a JSON Schema.
import json
response = client.chat.completions.create(
model="kimi-k3",
messages=[
{
"role": "user",
"content": (
"A deployment uses 8 B300 GPUs and 3 TB of NVMe. "
"Extract the configuration."
),
}
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "deployment_configuration",
"strict": True,
"schema": {
"type": "object",
"properties": {
"gpu_type": {"type": "string"},
"gpu_count": {"type": "integer"},
"storage_tb": {"type": "number"},
},
"required": [
"gpu_type",
"gpu_count",
"storage_tb",
],
"additionalProperties": False,
},
},
},
)
configuration = json.loads(
response.choices[0].message.content or "{}"
)
print(configuration)
Parse only the final content field as structured output — the reasoning field is not constrained to the JSON schema.
49. Tool calling
A tool definition contains name, description and a JSON Schema for parameters.
tools = [
{
"type": "function",
"function": {
"name": "get_repository_file",
"description": "Read a file from a repository",
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Repository-relative file path",
}
},
"required": ["path"],
"additionalProperties": False,
},
},
}
]
The model does not execute the function. It produces a structured request. Your application must:
- validate the arguments
- authorise the operation
- execute the tool
- capture its result
- return a matching
toolmessage - call the model again
Never execute tool calls blindly against privileged infrastructure. Use schema validation, allowlists, path restrictions, command sandboxing, approval steps, timeouts and audit logs.
The current vLLM recipe notes that Kimi K3 may occasionally emit a tool-call format that does not match the parser — validation and retry behaviour matter.
50. Local deployment troubleshooting
50.1 Out-of-memory during startup
Possible causes: insufficient aggregate HBM; gpu-memory-utilization too high; full 1M context configured immediately; excessive static-cache allocation; wrong parallelism shape; visual-model buffers; kernel workspace needs.
Responses:
- reduce context length
- reduce concurrency
- use FP8 KV cache after accuracy validation
- lower memory fractions slightly if startup allocations fail
- increase GPU count
- use hardware with more HBM
- change parallelism strategy
50.2 Server starts but accepts few requests
The KDA recurrent-state pool may be limiting admission. Try lower-precision KDA state where supported, change state-cache strategy, reduce per-request state slots, increase attention tensor parallelism, disable unnecessary speculative-decoding state, or cap maximum running requests to a stable value.
50.3 Slow time to first token
Causes: extremely long prompts; low prefix-cache hit rate; slow model loading; insufficient prefill parallelism; network bottlenecks; visual preprocessing; repeated reprocessing of context.
Responses: preserve stable prefixes exactly; enable prefix caching; separate prefill and decode pools; use context parallelism; monitor cache hits; avoid unnecessary early system-prompt edits.
50.4 Slow generation after the first token
Possible causes: inefficient MoE kernel; cross-node tensor-parallel communication; poor expert-routing communication; large batch; speculative decoding with low acceptance; unsuitable all-to-all backend.
50.5 Distributed startup hangs
Check: same model revision on every node; unique node ranks; reachable master address/port; NCCL/Gloo on the intended NIC; firewalls permit required ports; matching GPU drivers and container versions; correct clocks/DNS; functioning InfiniBand or RDMA.
50.6 Tool calls appear as text
Ensure the server includes:
--enable-auto-tool-choice
--tool-call-parser kimi_k3
Then confirm the engine version exposes the call in message.tool_calls.
50.7 Reasoning appears inside final content
Ensure:
--reasoning-parser kimi_k3
Then inspect both message.reasoning_content and message.content.
51. Production deployment checklist
Infrastructure
- Sufficient HBM for weights, states, caches and buffers
- High-speed local NVMe
- InfiniBand, RDMA or supported high-speed interconnect
- Redundant model-serving nodes
- Capacity and queue management
- GPU and network monitoring
Security
- Pinned model and container revisions
- Restricted
trust_remote_code - Container isolation
- No unrestricted terminal execution
- Least-privilege tool credentials
- Network egress controls
- Secret management
- Signed or scanned images
- Audit logging
Reliability
- Health checks
- Request timeouts
- Retry policies
- Tool-call schema validation
- Cache-failure handling
- Node-failure recovery
- Load shedding
- Context-length limits
- Rollback procedures
AI governance
- Workload-specific evaluations
- Hallucination testing
- Tool-action approval boundaries
- Human review for high-impact decisions
- Data-retention controls
- Reasoning-log policy
- Copyright and licence review
- Bias and safety testing
Performance
Measure separately:
- time to first token
- inter-token latency
- tokens per second
- cache-hit rate
- prefill throughput
- decode throughput
- GPU utilisation
- expert-load balance
- network bandwidth
- maximum stable concurrency
52. What Kimi K3’s architecture means strategically
Kimi K3’s importance is not simply that it has 2.8 trillion parameters. Its architecture combines:
- recurrent long-context processing
- periodic global attention
- depth-wise representation retrieval
- highly sparse expert computation
- native visual processing
- quantisation-aware deployment
- reinforcement learning for agentic workflows
| Component | Strategic role |
|---|---|
| KDA | Information flow through long sequences |
| Attention Residuals | Information flow through deep networks |
| Stable LatentMoE | Information flow across a huge expert parameter space |
| MoonViT-V2 | Visual information in the same backbone |
| Agentic RL | Extended action and verification loops |
The resulting model is intended as a foundation for persistent AI systems that can reason, observe, use tools, manipulate software, inspect visual results and continue working over long sessions.
Conclusion
Kimi K3 is a 2.8-trillion-parameter sparse multimodal model, not a conventional dense chatbot.
Main architectural innovations:
- Kimi Delta Attention — efficient recurrent state for long sequences
- Gated MLA — periodic global token-to-token interaction
- Attention Residuals — selective retrieval of earlier block representations
- Stable LatentMoE — activate 16 of 896 experts in a compressed latent space
- MoonViT-V2 — images and video frames into backbone-ready representations
- SiTU-GLU, RMSNorm and Quantile Balancing — stability at extreme scale and sparsity
- MXFP4/MXFP8 quantisation-aware training — more practical deployment
Built through:
- filtered multimodal pre-training
- architecture and scaling-law experiments
- progressive context extension from 8K to 1M
- supervised agentic fine-tuning
- RL across reasoning, general-agent and coding domains
- multi-effort policy training
- multi-teacher policy consolidation
- deployment-aware quantisation
Using Kimi K3 from a laptop is straightforward through the hosted API. Running the complete model on premises is a data-centre-scale undertaking. Current documented configurations range from eight very-high-memory B300 or MI35x accelerators to 16 H200/B200 GPUs or 32 H100 GPUs — plus fast storage, high-speed interconnects and specialised serving engines.
Correct interpretation of “open weights”:
Kimi K3 can be privately deployed and modified, but operating the complete frontier-scale model requires serious distributed-systems engineering and substantial accelerator infrastructure.
Primary technical references
- Moonshot Kimi K3 API quickstart and usage contract
- Kimi K3 technical report
- Official model repository and model card
- Current Kimi K3 vLLM deployment recipe
- Current Kimi K3 SGLang deployment cookbook
Discussion
Comments
Share feedback or questions about this page. No account required.
Loading comments…