DeepSeek: Architecture, Training and Deployment

DeepSeek is less a single model than a lineage of techniques that reshaped how the rest of the open-weight field builds MoE models: Multi-head Latent Attention (MLA), DeepSeekMoE, auxiliary-loss-free load balancing, multi-token prediction (MTP), GRPO reinforcement learning, and low-precision training at frontier scale. The current flagship generation, DeepSeek V4, ships as V4-Pro (1.6T total / 49B active) and V4-Flash (284B total / 13B active), both with 1,048,576-token context and thinking/non-thinking modes.

Executive summary

DeepSeek's real significance is architectural influence, not any single release. MLA, DeepSeekMoE-style fine-grained expert routing, and GRPO have been adopted or echoed across most of the open-weight field covered in this section. For most teams starting today, the practical entry point is still V3 plus R1 (well-documented, broadly supported by serving engines), while V4 — described in its June 2026 paper (arXiv:2606.19348) — pushes further with compressed sparse attention, hyper-connections and the Muon optimiser at million-token context.

When to use this page

• Understanding MLA, DeepSeekMoE and GRPO before evaluating any other MoE reasoning model in this section — these ideas recur everywhere.
• Deciding between DeepSeek V3+R1 (mature tooling) and V4 (newer, more capable, less battle-tested) for a self-hosted deployment.
• Choosing between V4-Pro (maximum capability) and V4-Flash (lower latency/cost) for a given workload.
• Pair with Kimi K3 (which uses a KDA/Gated-MLA descendant of these ideas), Open source models and the AI model landscape.

---

1. What kind of model is DeepSeek?

A lineage of open-weight, decoder-based, sparse MoE reasoning models whose architectural techniques (MLA, DeepSeekMoE, GRPO) have been widely adopted across the open-weight ecosystem.

1.1 Open-weight, practically documented

DeepSeek publishes weights, technical reports and — for several releases — inference code on GitHub, alongside detailed arXiv papers. This combination of weights and unusually thorough technical writing is a major reason DeepSeek's ideas propagated so quickly into other labs' architectures; it is still, however, open-weight, not a fully reproducible open-source release with published training data.

1.2 The generational arc

Generation	Headline contribution
V2	Introduced Multi-head Latent Attention (MLA) at scale
V3	DeepSeekMoE fine-grained experts, aux-loss-free balancing, MTP, FP8 training
R1	Large-scale GRPO reinforcement learning for reasoning, distilled into smaller dense models
DeepSeekMath	Originated GRPO (Group Relative Policy Optimisation)
Coder-V2	Code-specialised MoE built on the V2 architecture
V4	Compressed sparse attention, hyper-connections, Muon optimiser, 1M context, Pro/Flash split

1.3 V4-Pro and V4-Flash

Variant	Total params	Active params	Context	Positioning
V4-Pro	1.6T	49B	1,048,576	Maximum reasoning and agent capability
V4-Flash	284B	13B	1,048,576	Lower latency and cost, same context ceiling

Both support thinking and non-thinking operation, following the same reasoning-effort switch pattern now standard across the field.

---

2. Model configuration

Component	DeepSeek V4-Pro	DeepSeek V4-Flash
Architecture	Sparse MoE, compressed sparse attention	Sparse MoE, compressed sparse attention
Total parameters	1.6 trillion	284 billion
Activated parameters	49 billion	13 billion
Maximum context	1,048,576 tokens	1,048,576 tokens
Reasoning modes	Thinking / non-thinking	Thinking / non-thinking
Optimiser (training)	Muon (per V4 paper)	Muon (per V4 paper)

For historical/architectural reference, V3's headline configuration (671B total / 37B active) remains the most widely deployed and documented DeepSeek release across serving engines as of mid-2026.

---

3. The core techniques (read these before anything else)

3.1 Multi-head Latent Attention (MLA)

Standard multi-head attention caches full per-head key and value vectors for every previous token — the dominant memory cost at long context. MLA, introduced in V2 (arXiv:2405.04434), instead compresses each token's key/value information into a much smaller latent vector, stored in the cache, and reconstructs the full per-head representation only when attention is actually computed:

This single idea is why DeepSeek models can serve meaningfully longer contexts at lower memory cost than an equivalent dense-KV design, and it directly informed Kimi K3's Gated MLA (see Kimi K3).

3.2 DeepSeekMoE: fine-grained experts

DeepSeekMoE (introduced pre-V3, refined through V3) splits feed-forward capacity into a larger number of smaller experts than earlier MoE designs, plus one or more shared experts that process every token unconditionally:

Finer granularity gives the router more precise combinations to choose from per token than a smaller number of large experts would allow, improving specialisation without proportionally increasing active compute. This shared/routed split is now the default MoE pattern across almost every model in this guide.

3.3 Auxiliary-loss-free load balancing

Older MoE training added an auxiliary load-balancing loss term to the training objective to discourage the router from collapsing onto a few popular experts — but that extra loss term can itself distort the primary training objective. DeepSeek V3 instead uses a bias-based, auxiliary-loss-free balancing mechanism: each expert has a learned bias adjusted directly based on its recent load, nudging routing decisions without adding a competing loss term to gradient descent. The approach is conceptually close to (and predates in publication terms) the Quantile Balancing method Kimi K3 later used for the same problem.

3.4 Multi-token prediction (MTP)

DeepSeek V3 pretrains with an MTP module that predicts more than one future token per step, which both improves training signal density and can be repurposed at inference time as a draft model for speculative decoding — the same underlying idea Kimi K3, Gemma 4 and Nemotron 3 each adopted in their own forms.

3.5 GRPO (Group Relative Policy Optimisation)

GRPO, originating in DeepSeekMath (arXiv:2402.03300), is a reinforcement-learning method that estimates the advantage of an action by comparing it against a group of sampled outputs for the same prompt, rather than training a separate value/critic network as in classic PPO:

Removing the critic network reduces training memory and complexity considerably at the scale DeepSeek operates at, and GRPO (or close variants) is now used well beyond DeepSeek's own models across the field's RL-for-reasoning pipelines.

3.6 Low-precision and compressed-attention training (V4)

The V4 paper (arXiv:2606.19348) documents further pushes on top of the V3 recipe:

• Compressed sparse attention — extending the MLA lineage toward sparser, more compressed attention patterns for million-token efficiency.
• Hyper-connections — a residual-connection generalisation intended to improve gradient flow and representation diversity across very deep networks, addressing the same "dilution across many layers" problem Kimi K3's Attention Residuals target with a different mechanism.
• Muon optimiser — the same matrix-orthogonalising optimiser family later adopted (in a per-head variant) by Kimi K3.
• Million-token training — a progressive long-context curriculum consistent with the pattern used across the field.

---

4. Recommended reading order

For an engineer building genuine architectural literacy rather than skimming a model card, DeepSeek's own paper sequence is the clearest path through the ideas that now underpin most of the open-weight MoE field:

1. DeepSeekMoE — fine-grained expert routing and the shared/routed split.
2. DeepSeek-V2 (MLA) — arXiv:2405.04434 — compressed latent KV cache.
3. DeepSeek-V3 — arXiv:2412.19437 — aux-loss-free balancing, MTP, FP8 training at scale.
4. DeepSeekMath (GRPO) — arXiv:2402.03300 — the RL method almost everyone reasoning-tunes with now.
5. DeepSeek-R1 — arXiv:2501.12948 — large-scale RL-for-reasoning and distillation into smaller dense models.
6. DeepSeek-V4 — arXiv:2606.19348 — compressed sparse attention, hyper-connections, Muon, million-token scale.

Reading in this order shows the ideas accreting one at a time rather than trying to reverse-engineer six papers' worth of technique from a single V4 read.

---

5. Training and post-training

5.1 Pretraining and FP8

V3 was trained with FP8 mixed-precision at frontier scale, a notable engineering achievement given the numerical-stability challenges of low-precision training over trillions of tokens — requiring careful scaling strategies to avoid gradient and activation overflow/underflow. V4 continues pushing low-precision and compressed-representation training further per its paper.

5.2 Cold-start SFT before RL

Following the pattern also used by Kimi K3, R1's RL stage builds on a supervised cold-start policy rather than applying RL to a raw base model — giving the RL process a competent starting point for instruction-following, formatting and basic tool use before optimising for harder reasoning rewards.

5.3 Distillation into dense models

DeepSeek distilled R1's reasoning behaviour into smaller dense models (at various parameter scales), demonstrating that GRPO-trained reasoning behaviour from a large MoE teacher can meaningfully transfer into far smaller, easier-to-deploy dense students — a pattern since echoed by the wider field's teacher-to-student distillation pipelines.

---

6. Deployment

6.1 Practical starting point: V3 + R1

For most teams, V3 and R1 remain the most practical entry point into the DeepSeek lineage: broader serving-engine support, more community tooling, more third-party quantisations, and a larger base of deployment war stories than the newer V4 generation.

6.2 V4 hardware reality

Both V4-Pro (1.6T/49B) and V4-Flash (284B/13B) require multi-GPU infrastructure:

Model	Realistic minimum
V4-Flash (284B/13B)	Multi-GPU node (comparable class to Llama 4 Maverick)
V4-Pro (1.6T/49B)	Multi-node, high-memory cluster (comparable class to Kimi K3)

As always, total parameters drive storage/sharding requirements while active parameters drive per-token compute — a 1.6T-total model needs the whole checkpoint addressable across the cluster regardless of only 49B computing per token.

6.3 Serving engines

Engine	Notes
vLLM	Mature MLA and DeepSeekMoE kernel support; widely used for V3/R1
SGLang	RadixAttention prefix caching; strong MLA-aware KV-cache handling
DeepSeek's own inference repositories	Reference implementations released alongside V3/R1 on GitHub

6.4 Example: vLLM server for V3 (mature, well-documented path)

export HF_TOKEN="your-hugging-face-token"

docker run --rm \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-openai:latest \
  deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.9

6.5 Calling the endpoint with reasoning separated (R1-style)

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1",
    messages=[{"role": "user", "content": "Prove that the sum of the first n odd numbers is n^2."}],
    max_tokens=4096,
)

message = response.choices[0].message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)

Preserve the complete assistant message (not just content) across multi-turn conversations, the same requirement documented for Kimi K3 — reasoning-model serving contracts generally expect the full prior turn, not a truncated summary.

6.6 Multi-node example for V4-Pro

export HEAD_IP="10.0.0.10"
export IFACE_NAME="ib0"
export HF_TOKEN="your-hugging-face-token"

docker run --rm \
  --gpus all \
  --ipc=host \
  --network=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="$HF_TOKEN" \
  -e NCCL_SOCKET_IFNAME="$IFACE_NAME" \
  vllm/vllm-openai:latest \
  deepseek-ai/DeepSeek-V4-Pro \
  --trust-remote-code \
  --tensor-parallel-size 16 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.9

6.7 Choosing between V4-Pro and V4-Flash

Consideration	V4-Pro (1.6T/49B)	V4-Flash (284B/13B)
Priority	Maximum reasoning/agent capability	Lower latency and cost
Infrastructure	Multi-node, high-memory cluster	Single multi-GPU node, comparable to Llama 4 Maverick
Good fit	Complex agentic coding, deep research	High-volume chat, latency-sensitive agents

As with every Pro/Flash-style split in this section (compare Gemma 4's size ladder or Command A+'s single-tier design), the right choice depends on measuring cost per successfully completed task for your specific workload rather than defaulting to the larger model.

---

7. Common misconceptions

7.1 "DeepSeek is one model"

DeepSeek is better understood as a research lineage spanning V2, V3, R1, DeepSeekMath, Coder-V2 and V4, each contributing distinct, separately publishable ideas (MLA, DeepSeekMoE, GRPO, MTP) rather than one continuously versioned checkpoint. Citing "DeepSeek" without specifying which generation and variant is a common source of confusion in technical discussions.

7.2 "V4 makes V3 and R1 obsolete"

Not necessarily for a self-hosting decision. V4 is newer and reports stronger capability, but V3 and R1 have a longer track record of serving-engine support, community quantisations and deployment war stories. Treat "which generation" as a separate decision from "which is more capable on paper" — production readiness and capability are different axes.

7.3 "GRPO removes the need for reward models"

GRPO removes the need for a separate critic/value network, not for reward signals themselves. Rewards (test pass/fail, correctness checkers, reward models, human preference) are still required to score the group of sampled responses that GRPO compares against each other — GRPO changes how the advantage is estimated, not where the reward comes from.

7.4 "FP8 or lower-precision training means lower-quality weights"

Native low-precision training (V3's FP8, V4's further compressed-precision work) is a deliberate engineering choice validated against higher-precision baselines during development, not an accuracy compromise forced by resource constraints. When done well — with the numerical-stability techniques DeepSeek documents — it can match higher-precision training quality at substantially lower training cost.

---

8. Troubleshooting local deployment

8.1 Out-of-memory despite MLA's compressed KV cache

MLA reduces KV-cache memory relative to standard multi-head attention, but does not eliminate it, and does not reduce the memory required to store the full MoE checkpoint. Confirm whether an out-of-memory error is occurring during weight loading (a total-parameter/sharding problem) versus during long-context generation (a KV-cache sizing problem) before choosing a fix.

8.2 Uneven GPU utilisation across an expert-parallel cluster

Persistent load imbalance despite aux-loss-free balancing can indicate the balancing mechanism's bias terms have not yet converged for your specific workload's token distribution (which may differ from DeepSeek's own training distribution), or that expert-parallel topology does not match the actual GPU/interconnect layout. Monitor per-expert load directly rather than assuming balancing is automatically optimal for every deployment.

8.3 Reasoning trace missing or duplicated in R1 responses

Confirm the serving engine's reasoning parser version matches the deployed R1 checkpoint's expected output format — this is the same class of chat-template/parser mismatch that affects every reasoning model in this section after an engine or model upgrade.

8.4 Slower-than-expected decode on V4 at long context

Confirm compressed sparse attention (V4's mechanism) is actually enabled and supported by the serving engine version in use; falling back silently to standard attention at long context can produce correct but much slower generation, particularly noticeable only once requests approach the million-token range.

---

9. Licence

DeepSeek's model licences have varied across releases and should be checked per model on Hugging Face at download time; several DeepSeek releases use permissive MIT-style code licences alongside model-specific weight licences. Do not assume V3, R1 and V4 all carry identical terms — verify each checkpoint's licence individually before commercial deployment.

---

10. Engineer reading checklist

• MLA: compressed latent KV cache vs full per-head KV cache
• DeepSeekMoE: fine-grained experts plus shared experts
• Auxiliary-loss-free load balancing via per-expert bias adjustment
• GRPO: group-relative advantage estimation without a critic network
• Multi-token prediction as both a training signal and a speculative-decoding draft head
• Why V3+R1 remains the pragmatic default while V4 is newer and less battle-tested
• Total vs active parameters for V4-Pro (1.6T/49B) and V4-Flash (284B/13B)

---

References

• DeepSeek-V4 — arXiv:2606.19348 (compressed sparse attention, hyper-connections, Muon, million-token context)
• DeepSeek-V3 — GitHub repository and arXiv:2412.19437
• DeepSeek-R1 — GitHub repository and arXiv:2501.12948
• DeepSeek-V2 (MLA) — arXiv:2405.04434
• DeepSeekMath (GRPO) — arXiv:2402.03300
• DeepSeek-Coder-V2 — arXiv:2406.11931

Key takeaways

• DeepSeek's lasting impact is architectural: MLA, DeepSeekMoE and GRPO now appear, in some form, across most of the models covered in this section.
• V4-Pro (1.6T/49B) targets maximum capability; V4-Flash (284B/13B) targets lower latency and cost, both at 1M context.
• V3 + R1 remains the more mature, better-tooled starting point for most self-hosted deployments today.
• Total parameters govern storage and sharding; active parameters govern per-token compute — plan infrastructure around both.

What to do next

Continue to Kimi K3 to see MLA's ideas extended into Gated MLA and KDA, browse Open source models, or return to the AI model landscape for full category-wide selection guidance.