NVIDIA Nemotron 3: Architecture, Training and Deployment

Nemotron 3 is NVIDIA's open-weight family built around a hybrid Mamba-2 + attention + LatentMoE architecture, released with an unusually complete package: base and post-trained checkpoints, datasets, training recipes, and cookbooks for TensorRT-LLM, vLLM and SGLang. The primary documented release, Nemotron 3 Super, has 120B total / 12B active parameters, 1,048,576-token context, multi-token prediction, and was natively pretrained in NVFP4. A larger sibling, Nemotron 3 Ultra (550B total / 55B active), is referenced in NVIDIA's broader model family positioning.

Executive summary

Nemotron 3's defining trait is not any single benchmark score — it is how much of the development pipeline NVIDIA publishes alongside the weights: datasets, training recipes and reward-model-related components, not just a final instruction-tuned checkpoint. Architecturally, its Mamba-2 + attention hybrid targets the same long-context efficiency problem as Kimi K3's KDA and Qwen 3.5's Gated DeltaNet, but is trained natively in NVFP4 — a 4-bit format designed into the pretraining process itself rather than applied only as a post-training quantisation step.

When to use this page

• Selecting a model where training recipes and datasets (not just weights) matter for research, auditing or fine-tuning.
• Understanding Mamba-2 + attention hybrids and native NVFP4 pretraining.
• Choosing between Super (120B/12B, primary documented release) and the larger Ultra sibling.
• Pair with Kimi K3 and Qwen 3.5 for two other hybrid-attention approaches, plus Open source models and the AI model landscape.

---

1. What kind of model is Nemotron 3?

Highly open (weights, datasets, recipes), decoder-based, hybrid Mamba-attention Mixture-of-Experts family natively pretrained in 4-bit precision.

1.1 More open than "open-weight"

Most models in this section publish final weights plus a technical report. NVIDIA's Nemotron releases go further, publishing:

• Base checkpoints.
• Post-trained (instruction-tuned) checkpoints.
• Quantised checkpoints.
• Training datasets.
• Post-training data.
• Model recipes (the actual training configuration/pipeline, not just a paper description of it).
• Reward-model-related components used in RL post-training.

This does not make Nemotron 3 a fully OSI-style open-source model — it is still, correctly, an open-weight release — but the amount of published development material is substantially higher than most peers in this guide, and matters directly for teams doing research, reproducibility auditing, or deep fine-tuning rather than only inference.

1.2 Super and Ultra

Variant	Total params	Active params	Status in this guide
Nemotron 3 Super	120B	12B	Primary documented release covered here
Nemotron 3 Ultra	550B	55B	Larger sibling referenced in NVIDIA's model family positioning

Treat Super as the primary, fully documented release with concrete deployment cookbooks; treat Ultra as the larger sibling in the same architectural family when a workload genuinely needs more total capacity than Super provides.

---

2. Model configuration

Component	Nemotron 3 Super
Architecture	Hybrid Mamba-2 + attention + LatentMoE
Total parameters	120 billion
Activated parameters	12 billion
Maximum context	1,048,576 tokens
Speculative decoding	Multi-token prediction (MTP)
Pretraining precision	NVFP4 (native)
RL rollouts	~1.2 million, across 21 environments
Deployment recipes	TensorRT-LLM, vLLM, SGLang cookbooks
Training stack	NeMo, Megatron-LM

---

3. Architecture

3.1 Mamba-2: a state-space alternative to attention

Mamba-2 is a state-space model layer: rather than comparing a token against every previous token (attention's approach), it maintains a compressed recurrent state updated as the sequence is processed, with the update rule and state size designed for hardware-efficient parallel computation despite being fundamentally recurrent:

State S_t = f(S_{t-1}, token_t)   # selective, input-dependent update
Output_t = g(S_t, token_t)

Mamba-2's "selective" mechanism means the update itself depends on the current input, letting the model learn what to retain and what to discard per step — conceptually related to KDA's channel-wise retention and Gated DeltaNet's gating, though state-space models and linear-attention-family recurrent layers arrive at similar goals via different mathematical formulations.

3.2 Hybrid Mamba-2 + attention

Pure state-space stacks share the same limitation as pure linear-attention stacks: a compressed recurrent state cannot preserve every exact token-to-token relationship as reliably as full attention can. Nemotron 3 Super interleaves Mamba-2 layers with conventional attention layers, following the now-standard hybrid pattern across this section:

Layer type	Role
Mamba-2	Efficient long-sequence state carrying; near-linear cost
Full attention	Periodic global retrieval, precise long-range dependencies
LatentMoE	Specialised, compressed-latent expert feed-forward processing

3.3 LatentMoE

Rather than routing tokens directly to full-width experts, Nemotron 3's LatentMoE first projects the token representation into a smaller latent dimension before routing to experts operating in that compressed space, then projects back up afterward — the same general shape as Kimi K3's Stable LatentMoE:

This reduces the communication and computation cost of the routed-expert path relative to running full-width experts directly, which matters more as expert count and total parameters grow.

3.4 Native NVFP4 pretraining

Most models in this section apply low-precision quantisation after training (or, at best, quantisation-aware fine-tuning during post-training, as Kimi K3 does for MXFP4). Nemotron 3 Super is natively pretrained in NVFP4 — NVIDIA's 4-bit floating-point format — from the start of large-scale pretraining, not merely compressed afterward.

Training natively at 4-bit precision from the beginning requires the training recipe itself (scaling strategies, numerical stability techniques, gradient handling) to be designed around the reduced dynamic range rather than treating quantisation as a downstream compression step applied to weights that were never trained with that constraint in mind. This is a meaningfully different engineering approach from quantisation-aware post-training and is one of Nemotron 3's more distinctive technical claims.

3.5 Multi-token prediction

Nemotron 3 Super includes an MTP component for native speculative decoding, consistent with DeepSeek V3, Kimi K3, Gemma 4 and GLM-5.2's adoption of the same general mechanism: a lightweight draft head proposes multiple future tokens, verified by the main model in a single pass.

---

4. Training and post-training

4.1 Multi-environment reinforcement learning

Nemotron 3 Super's post-training includes reinforcement learning across approximately 1.2 million rollouts spanning 21 distinct environments — a substantially broader RL environment set than a single coding-or-maths reward signal, intended to build generalised agentic and reasoning capability across varied task types rather than over-specialising on one domain.

4.2 NeMo and Megatron-LM

NVIDIA's own NeMo framework and Megatron-LM provide the training infrastructure underlying Nemotron 3's pretraining and post-training — both are themselves published, open frameworks, further reinforcing the "recipes, not just weights" positioning described in Section 1.

---

5. Deployment

5.1 Hardware planning

At 120B total / 12B active, Nemotron 3 Super sits closer to Mistral Small 4 and Llama 4 Scout in overall deployment weight than to trillion-parameter flagships such as Kimi K3 or DeepSeek V4-Pro, though its native NVFP4 training means the published weights are already 4-bit-native rather than a post-hoc quantisation of a higher-precision original — a genuine memory advantage at deployment time if the serving stack fully supports NVFP4 inference.

Deployment goal	Suggested starting point
Development / evaluation	Single high-memory GPU at NVFP4/FP8
Production, moderate concurrency	Multi-GPU node
Nemotron 3 Ultra (550B/55B)	Multi-node cluster, comparable class to other large MoE flagships

5.2 Serving engines

Engine	Notes
TensorRT-LLM	NVIDIA's own high-performance inference engine; first-class NVFP4 and Nemotron-specific kernel support
vLLM	OpenAI-compatible; documented Nemotron cookbook
SGLang	RadixAttention prefix caching; documented Nemotron cookbook

5.3 Example: vLLM server

export HF_TOKEN="your-hugging-face-token"

docker run --rm \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-openai:latest \
  nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.9

5.4 Example: TensorRT-LLM (NVIDIA-native path)

trtllm-serve \
  nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
  --tp_size 4 \
  --max_seq_len 65536 \
  --kv_cache_free_gpu_memory_fraction 0.85 \
  --host 0.0.0.0 \
  --port 8000

TensorRT-LLM is the recommended path when maximum inference performance on NVIDIA hardware matters more than serving-engine portability across vendors — it is built and tuned by the same organisation that designed the NVFP4 format and Nemotron's kernels.

5.5 Calling the endpoint

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8",
    messages=[{"role": "user", "content": "Plan a migration from a monolith to microservices, step by step."}],
    max_tokens=4096,
)

print(response.choices[0].message.content)

5.6 Multi-node deployment for Nemotron 3 Ultra

export HEAD_IP="10.0.0.10"
export IFACE_NAME="ib0"
export HF_TOKEN="your-hugging-face-token"

docker run --rm \
  --gpus all \
  --ipc=host \
  --network=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="$HF_TOKEN" \
  -e NCCL_SOCKET_IFNAME="$IFACE_NAME" \
  vllm/vllm-openai:latest \
  nvidia/NVIDIA-Nemotron-3-Ultra \
  --trust-remote-code \
  --tensor-parallel-size 16 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.9

At Ultra's scale (550B/55B), treat infrastructure planning the same way as any other trillion-parameter-class flagship in this section: full-checkpoint sharding across the cluster, high-bandwidth interconnect for expert-parallel and Mamba-state exchange, and a conservative initial context-length ceiling.

5.7 Super vs Ultra decision guide

Consideration	Nemotron 3 Super (120B/12B)	Nemotron 3 Ultra (550B/55B)
Documentation depth	Primary documented release, concrete cookbooks	Referenced sibling, less granular public detail
Infrastructure	Single high-memory GPU to small multi-GPU node	Multi-node cluster, comparable to other frontier MoE flagships
Best fit	Teams wanting NVIDIA-native tooling at a moderate footprint	Workloads requiring maximum Nemotron-family capability

For most teams evaluating Nemotron 3 for the first time, Super is the more practical and better-documented starting point; move to Ultra only once a specific, measured capability gap justifies the additional infrastructure investment.

---

6. Common misconceptions

6.1 "Publishing datasets and recipes makes Nemotron 3 fully open source"

It makes Nemotron 3 substantially more transparent than most open-weight peers, but "fully open source" in the Open Source Initiative's sense also implies the freedoms to use, study, modify and share the complete system without restriction. Always check the specific licence attached to each published artefact (weights, dataset, code) individually rather than assuming the whole release qualifies uniformly.

6.2 "Mamba-2 layers make attention layers unnecessary"

Nemotron 3 Super keeps attention layers deliberately, for the same reason every hybrid model in this section does — a compressed recurrent state cannot reliably preserve every exact long-range dependency the way full attention can. The hybrid combination, not Mamba-2 alone, is the design that matters.

6.3 "Native NVFP4 pretraining means the model is inherently lower quality than a BF16-trained equivalent"

Native low-precision training is validated during development against higher-precision baselines; when the training recipe is properly designed for the reduced dynamic range (as NVIDIA documents for NVFP4), it is not simply "BF16 quality with rounding error." Treat quality claims as needing workload-specific validation either way, rather than assuming precision alone determines quality.

6.4 "Super and Ultra are interchangeable, just different sizes"

They share an architectural family, but Ultra's larger total and active parameter counts (550B/55B vs Super's 120B/12B) mean substantially different infrastructure requirements and, likely, different capability/cost trade-offs. Confirm which specific checkpoint a given claim or benchmark refers to before comparing.

---

7. Troubleshooting local deployment

7.1 TensorRT-LLM build or engine errors

Confirm the TensorRT-LLM version matches the version Nemotron 3's cookbook was validated against — NVFP4 and Nemotron-specific kernel support are actively developed, and version mismatches are the most common source of build failures for newly released model architectures.

7.2 Out-of-memory despite NVFP4's smaller footprint

Confirm the deployment is actually loading the native NVFP4 checkpoint and not accidentally falling back to a higher-precision variant; also account for KV-cache and Mamba-state memory separately from weight memory, since the recurrent state pool and attention KV pool are distinct allocations in a hybrid architecture.

7.3 Inconsistent behaviour across the 21 RL environments' task types

If evaluation results vary widely by task category, review which of the 21 training environments most closely resembles your production workload — broad multi-environment RL improves generalisation but does not guarantee uniform performance across every possible task category; targeted fine-tuning may still be warranted for a narrow, high-value use case.

7.4 Distributed training reproduction issues using published recipes

Given NVIDIA publishes actual recipes (not just descriptions), confirm the reproduction environment matches the documented NeMo/Megatron-LM versions and cluster topology assumptions — recipe-level reproduction is far more sensitive to exact software-stack versions than simply running inference against a published checkpoint.

---

8. Licence

Check the specific Nemotron 3 checkpoint's licence on Hugging Face and NVIDIA's model catalogue at download time — NVIDIA's open model releases have historically used NVIDIA-specific open model licences alongside more standard licences for particular components (datasets, code). Because Nemotron 3 publishes datasets and recipes in addition to weights, confirm licensing separately for each artefact type (weights, dataset, training code) before combining them in a commercial pipeline — they are not guaranteed to share identical terms.

---

9. Engineer reading checklist

• Mamba-2 state-space layers vs attention, and why Nemotron 3 hybridises both
• LatentMoE: compressed-latent expert routing, same shape as Kimi K3's Stable LatentMoE
• Native NVFP4 pretraining vs post-hoc or quantisation-aware post-training quantisation
• Why 21-environment, 1.2M-rollout RL aims for generalised rather than single-domain agentic capability
• What "published recipes and datasets" adds beyond a typical open-weight release
• TensorRT-LLM vs vLLM/SGLang trade-offs for NVIDIA-native deployment

---

References

• NVIDIA developer blog: Nemotron 3 Super announcement
• Hugging Face: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8
• NeMo framework documentation and repository
• Megatron-LM repository
• TensorRT-LLM documentation and Nemotron deployment cookbook

Key takeaways

• Nemotron 3 publishes datasets and training recipes alongside weights — more transparent than most open-weight releases, still short of fully open source.
• Super (120B/12B) is the primary documented release; Ultra (550B/55B) is the larger sibling for workloads needing more total capacity.
• Native NVFP4 pretraining is a distinct engineering approach from post-hoc quantisation, and yields genuinely 4-bit-native published weights.
• TensorRT-LLM is the NVIDIA-native performance path; vLLM and SGLang offer more portable alternatives with documented Nemotron cookbooks.

What to do next

Compare with Kimi K3 and Qwen 3.5 for two other hybrid-attention approaches, browse Open source models, or return to the AI model landscape for full category-wide selection guidance.