Llama 4: Architecture, Training and Deployment

Llama 4 is Meta's early-fusion multimodal Mixture-of-Experts family, released as two open-weight models: Maverick (400B total / 17B active, 128 experts, 1M-token context) and Scout (109B total / 17B active, 16 experts, 10M-token context). Both are distributed under Meta's own Llama Community Licence rather than a standard OSI-approved licence such as Apache 2.0.

Executive summary

Llama 4 matters less for raw parameter count and more for two things: it was Meta's first MoE-based Llama generation, and it uses early fusion — image and text tokens enter the same backbone from the very first layer rather than being combined later. Scout's headline feature is an extraordinarily long 10-million-token context on a comparatively modest 109B/17B configuration. The most important caveat for procurement and legal review: the Llama Community Licence is not Apache 2.0 and carries its own conditions.

When to use this page

• Evaluating Llama 4 Maverick or Scout for very-long-context or broad-ecosystem workloads.
• Understanding early-fusion multimodality vs the separate-encoder approach used elsewhere.
• Confirming licence obligations before commercial deployment or redistribution.
• Pair with Open source models, the AI model landscape and Kimi K3 for a contrasting MoE and licensing model.

---

1. What kind of model is Llama 4?

Open-weight, decoder-based, early-fusion multimodal Mixture-of-Experts family, released under a custom community licence.

1.1 Open-weight, but not Apache-licensed

This is the single most consequential fact for anyone evaluating Llama 4 for commercial use: unlike Qwen 3.5, Gemma 4, Mistral or GPT-OSS, Llama 4 is not released under Apache 2.0 or MIT. It uses the Llama Community Licence, a custom Meta licence that historically has included conditions such as:

• Attribution requirements ("Built with Llama").
• A monthly-active-user threshold above which a separate commercial licence must be negotiated with Meta.
• Restrictions on using Llama outputs to train competing foundation models.
• Region-specific restrictions in some releases.

None of this prevents most organisations from using Llama 4, but it does mean the licence must be read, not assumed to be equivalent to a standard open-source licence. Always check the current licence text at the point of download, since terms have changed across Llama generations.

1.2 Two models, two design points

Variant	Total params	Active params	Experts	Context	Multimodal
Llama 4 Scout	109B	17B	16	10,000,000 tokens	Native text + image
Llama 4 Maverick	400B	17B	128	1,048,576 tokens	Native text + image

Both share the same active parameter count (17B), but Maverick reaches far greater total capacity through many more, smaller experts, while Scout trades expert count for an exceptionally long context window on a much smaller total footprint.

1.3 Early-fusion multimodality

Llama 4 uses early fusion: image patches are tokenised and injected into the model's input sequence alongside text tokens from the first layer onward, rather than being processed by a separate vision tower and combined only at a later cross-attention or projection stage.

Early fusion lets the same attention and MoE machinery process text and visual tokens identically, which can improve cross-modal reasoning consistency, at the cost of a larger effective sequence length for image-heavy inputs (since images become many tokens rather than a compressed side-channel).

---

2. Model configuration

Component	Llama 4 Scout	Llama 4 Maverick
Architecture	Sparse MoE, early-fusion multimodal	Sparse MoE, early-fusion multimodal
Total parameters	109B	400B
Activated parameters	17B	17B
Routed experts	16	128
Maximum context	10,000,000 tokens	1,048,576 tokens
Modalities	Text, image	Text, image
Licence	Llama Community Licence	Llama Community Licence

2.1 Why Scout's context is so much longer than Maverick's

Scout's design deliberately favours context length over expert breadth: fewer, larger experts (16) and a smaller total parameter count leave more of the engineering budget — attention design, positional handling, long-context training data — directed at making a 10-million-token window usable, rather than at maximising total stored capacity. Maverick instead spends its parameter budget on breadth (128 experts) for stronger general capability at a more conventional (still very large) 1M-token ceiling.

2.2 Total vs active parameters, once more

Both models activate the same 17B parameters per token. The difference between them is how much total capacity sits behind that active compute: Maverick's 400B total (128 experts) gives the router a much larger pool of specialists to choose from than Scout's 109B total (16 experts), at the cost of a much larger checkpoint to store and shard.

---

3. Architecture

3.1 MoE routing at very different expert counts

Maverick's 128 experts and Scout's 16 experts represent two different points on the MoE sparsity spectrum described in the landscape guide: more, smaller experts (Maverick) generally allow finer-grained specialisation and higher total capacity per unit of active compute; fewer, larger experts (Scout) simplify routing and expert-parallel infrastructure, which can matter more when the priority is stable behaviour across an extremely long context rather than maximum total capacity.

3.2 Long-context engineering for Scout

Reaching a 10-million-token context is not simply a matter of increasing a configuration constant. It requires:

• Positional encoding and/or scaling strategies that remain stable far beyond the lengths seen during the bulk of pre-training.
• Attention or caching strategies that keep memory growth manageable at extreme lengths (full quadratic attention at 10M tokens is not practical without mitigation).
• Training data and evaluation tasks specifically constructed to require retrieval from very distant positions in the sequence, not just long documents that could be answered from the nearby context — the same synthetic long-context task design described for Kimi K3.

In practice, very few production workloads genuinely need 10 million tokens of context; Scout's headline number is best treated as a ceiling to plan infrastructure around, not a default operating point.

3.3 Multimodal token cost

Because Llama 4 uses early fusion, every image consumes real sequence-length budget as vision tokens rather than a fixed, small side-channel embedding. High-resolution or multi-image requests can consume a meaningful fraction of the context window — a capacity-planning detail that matters more for early-fusion models than for architectures using a heavily compressed visual token count (compare Kimi K3's MoonViT-V2 pixel-shuffle downsampling, which explicitly reduces visual token count by roughly 4×).

3.4 Comparison with other multimodal fusion strategies

Model	Fusion strategy	Visual token cost
Llama 4	Early fusion — raw image tokens in the shared sequence from layer 1	Higher — no dedicated compression stage
Kimi K3	Native training with a dedicated vision encoder (MoonViT-V2) plus pixel-shuffle downsampling	Lower — ~4× token reduction before entering the backbone
Gemma 4 (12B)	Encoder-free, direct patch projection	Depends on patch size and image resolution

Early fusion without an explicit compression stage trades higher per-image token cost for architectural simplicity — every image patch becomes a token processed identically to text, with no separate encoder or downsampling step to tune or maintain.

---

4. Training and post-training

Meta's Llama 3 technical report (arXiv:2407.21783) documents the general Llama pretraining and post-training pipeline that Llama 4 builds on: large-scale web, code and multilingual pretraining; supervised fine-tuning on curated instruction data; and preference optimisation. Llama 4 extends this lineage with:

• Native multimodal pretraining rather than a later-stage vision adapter, consistent with the early-fusion architecture.
• MoE-specific training infrastructure (expert parallelism, load-balancing losses) introduced for the first time in a mainline Llama release.
• Long-context curriculum training for Scout, following the now-standard progressive-length pattern (shorter sequences first, extreme lengths concentrated later in training) used across the field.

Meta has not published the same level of granular architectural detail for Llama 4 as it did in the Llama 3 report; treat Llama 4-specific claims as sourced from model cards and release documentation rather than a dedicated Llama 4 paper.

---

5. Deployment

5.1 Hardware reality

Neither model is a single-GPU proposition at full precision:

Model	Realistic minimum for self-hosting
Scout (109B/17B)	Multi-GPU node (e.g. 4–8 modern datacentre GPUs) at reduced precision; smaller than Maverick but still not single-GPU at full quality
Maverick (400B/17B)	Multi-GPU node minimum; multi-node cluster for production concurrency and full 1M context

As with every large MoE model in this guide, the full total-parameter checkpoint must be resident or shardable across the cluster even though only 17B parameters compute per token — sparse activation reduces compute, not storage.

5.2 Serving engines

Engine	Notes
vLLM	Documented Llama 4 support with tensor and expert parallelism
SGLang	RadixAttention prefix caching; multimodal request support
Llama Stack	Meta's own reference stack for building agents and applications on top of Llama models, including tool-calling and RAG scaffolding

5.3 Example: vLLM server for Scout

export HF_TOKEN="your-hugging-face-token"

docker run --rm \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-openai:latest \
  meta-llama/Llama-4-Scout-17B-16E \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.9

Note the --max-model-len is set far below Scout's 10M-token ceiling — start conservatively, benchmark, and only extend context length once the deployment's memory and latency budget has been validated at the target concurrency.

5.4 Calling the endpoint

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E",
    messages=[{"role": "user", "content": "Summarise the attached repository's architecture."}],
    max_tokens=2048,
)

print(response.choices[0].message.content)

5.5 Multi-node deployment for Maverick

Production Maverick deployments typically span multiple nodes:

export HEAD_IP="10.0.0.10"
export IFACE_NAME="ib0"
export HF_TOKEN="your-hugging-face-token"

docker run --rm \
  --gpus all \
  --ipc=host \
  --network=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="$HF_TOKEN" \
  -e NCCL_SOCKET_IFNAME="$IFACE_NAME" \
  vllm/vllm-openai:latest \
  meta-llama/Llama-4-Maverick-17B-128E \
  --trust-remote-code \
  --tensor-parallel-size 16 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.9

Maverick's 128-expert configuration makes expert-parallel communication a first-order concern at multi-node scale — confirm the interconnect is high-bandwidth (InfiniBand or equivalent RDMA-capable Ethernet) before committing to a tensor-parallel topology spanning nodes.

5.6 Why Llama remains relevant despite the licence friction

Even with the Llama Community Licence's extra diligence requirement, Llama 4 retains real advantages over some Apache 2.0 alternatives:

• Ecosystem maturity — the largest body of third-party fine-tunes, quantisations and deployment guides of any open-weight family.
• Cloud availability — pre-integrated support across most major cloud AI platforms and managed inference providers.
• Tooling — Llama Stack and a wide range of community agent frameworks built specifically against Llama's tool-calling conventions.
• Familiarity — many engineering teams already have Llama-specific operational experience from earlier generations (Llama 2, Llama 3).

Whether that ecosystem advantage outweighs the licence friction depends on the specific deployment scale and product — for a small internal tool unlikely to approach the licence's usage threshold, ecosystem maturity may dominate; for a large consumer-facing product, the licence review deserves to happen before architecture decisions, not after.

5.7 Building an agent with Llama Stack

Llama Stack provides standardised APIs for inference, tool-calling, RAG and safety shields on top of a running Llama 4 endpoint, intended to reduce the amount of bespoke agent-harness code needed compared to wiring a raw OpenAI-compatible client directly into an application.

---

6. Common misconceptions

6.1 "Llama 4 is Apache 2.0 like most other open models"

No. This is the most consequential misconception to correct early — Llama 4 uses the custom Llama Community Licence, which has historically included a monthly-active-user threshold, attribution requirements and restrictions on using outputs to train competing models. Treat this as a distinct legal review item, not a formality.

6.2 "Scout's 10M context means it will reliably use information from 10M tokens back"

Maximum accepted context length and effective retrieval quality are different properties, a distinction that applies to every long-context model in this section. A model can accept a 10-million-token input without reliably locating and using a specific fact planted near the beginning of it. Validate retrieval quality at the context lengths you actually intend to use, rather than assuming the documented ceiling behaves uniformly well throughout its range.

6.3 "128 experts (Maverick) means the model is 8× 'smarter' than Scout's 16 experts"

Expert count is a capacity and specialisation lever, not a linear intelligence multiplier. Maverick's larger expert pool gives the router more specialised combinations to draw from, which can improve quality on complex tasks, but is not a proxy for a specific fixed capability multiple over Scout.

6.4 "Since both share 17B active parameters, they cost the same to run"

Active parameters govern per-token compute, but Maverick's much larger total parameter count (400B vs 109B) increases memory footprint, sharding complexity, and often network overhead from a larger expert-parallel topology — all of which affect total infrastructure cost even at matched active-parameter compute.

---

7. Troubleshooting local deployment

7.1 Out-of-memory despite "only" 17B active parameters

This is the single most common Llama 4 deployment mistake: sizing GPU memory around the active-parameter figure instead of the full total-parameter checkpoint. Both Scout (109B total) and especially Maverick (400B total) require the complete checkpoint addressable across the cluster regardless of active compute per token.

7.2 Licence-compliance checks failing procurement review

If a deployment is blocked at legal/procurement review, the most common cause is treating Llama 4 as if it were Apache 2.0-licensed during initial planning. Re-run the review against the actual current Llama Community Licence text, and check specifically for the monthly-active-user threshold and any output-usage restrictions relevant to the intended product.

7.3 Slow time-to-first-token at very long context

For Scout specifically, verify prefix caching is enabled and that the request's system prompt and shared context are stable across calls — recomputing a multi-million-token prefix from scratch on every request is the dominant cost driver at extreme context lengths, far more than raw model size.

7.4 Multimodal requests degrading throughput unexpectedly

Because Llama 4 uses early fusion, image-heavy requests consume real sequence-length budget as vision tokens rather than a small fixed side-channel. A sudden throughput drop when image traffic increases is often simply a context-length and batching effect, not a bug — plan capacity assuming images meaningfully extend effective sequence length.

---

8. Licence

The Llama Community Licence is the single biggest differentiator versus most other models in this section. Before any commercial deployment:

1. Read the current licence text from llama.com/docs at download time — terms have changed between Llama generations.
2. Check for a monthly-active-user threshold that would require a separate commercial agreement with Meta.
3. Check attribution requirements for any user-facing product built on Llama 4.
4. Check restrictions on using Llama 4 outputs to train or improve other foundation models.
5. Confirm the licence permits your target region and use case.

This is materially more legal diligence than Apache 2.0-licensed alternatives such as Qwen 3.5, Gemma 4, Mistral or GPT-OSS require, and should be factored into build-vs-buy and model-selection decisions early, not discovered late in a procurement cycle.

---

9. Engineer reading checklist

• Why Llama 4's licence review is a distinct, mandatory step — it is not Apache 2.0
• Early fusion vs separate-encoder multimodality, and its effect on effective sequence length
• Why Scout and Maverick share 17B active parameters but differ hugely in total capacity and context length
• Expert count (16 vs 128) as a design lever independent of active-parameter count
• Why a 10M-token ceiling is a planning number, not a default operating context length
• Llama Stack's role as an agent/tool/RAG layer on top of raw inference

---

References

• llama.com/docs and llama.com/downloads
• Hugging Face: meta-llama/Llama-4-Maverick-17B-128E
• GitHub: github.com/meta-llama/llama-models
• Llama 3 Technical Report — arXiv:2407.21783 (pretraining/post-training pipeline lineage)
• Llama Stack documentation and repository

Key takeaways

• Llama 4 uses the Llama Community Licence, not Apache 2.0 — read it before committing to commercial use.
• Maverick (400B/17B, 128 experts) and Scout (109B/17B, 16 experts, 10M context) are two different design points sharing the same active-parameter budget.
• Early fusion means images consume real context-window budget as tokens, not a small fixed side-channel.
• Treat Scout's 10M-token ceiling as an infrastructure planning limit, not a routine operating context length.

What to do next

Compare licensing with Qwen 3.5 or Gemma 4 (both Apache 2.0), or return to the AI model landscape for full category-wide selection guidance.