Gemma 4: Architecture, Training and Local Deployment

Gemma 4 is Google DeepMind's Apache 2.0 open-weight family aimed squarely at local, edge and cost-efficient server deployment. The suite spans dense and Mixture-of-Experts designs from roughly 2.3 billion to 31 billion parameters, adds a genuinely encoder-free multimodal design at 12B, and includes an MoE model (26B-A4B) alongside a dense flagship (31B).

Executive summary

Gemma 4 is Google's answer to "what is the best model I can run on a laptop, phone or single GPU today?" rather than a frontier-scale trillion-parameter play. Its architecture choices — local/global hybrid attention, KV-cache reduction, quantisation-aware training, and (at 12B) an encoder-free multimodal design — are all aimed at cutting the memory and latency cost of running a capable model outside a datacentre. Apache 2.0 licensing removes the usual open-weight commercial friction.

When to use this page

• Selecting a local or on-device model for offline assistants, private document processing or mobile agents.
• Understanding local/global sliding-window attention and why it shrinks the KV cache.
• Evaluating quantisation-aware training (QAT) vs post-hoc quantisation for edge deployment.
• Pair with Open source models, the AI model landscape and Qwen 3.5 for a larger-scale hybrid-attention comparison.

---

1. What kind of model is Gemma 4?

Open-weight (Apache 2.0), decoder-based, multimodal family spanning dense and sparse MoE designs, optimised for local and edge inference rather than frontier scale.

1.1 Open-weight, Apache 2.0 — and genuinely local-first

Gemma 4 ships under Apache 2.0, the same permissive licence family as Qwen 3.5, Mistral Small 4 and GPT-OSS. Google DeepMind additionally publishes a Responsible GenAI Toolkit and model cards describing intended use, evaluation results and known limitations — useful inputs for an internal AI-governance review, though they do not amount to a fully open, reproducible training pipeline. As with every model in this section, treat Gemma 4 as open-weight, not OSI-style fully open source.

1.2 A family spanning four size classes

Variant	Design	Approx. size	Target
E2B / E4B	Dense, per-layer embeddings, frozen compact vision/audio encoders	~2.3B / ~4.5B effective	On-device, phones, laptops
12B	Dense, encoder-free multimodal (trained from scratch)	12B	Local multimodal workstation use
26B-A4B	Sparse MoE	~26B total / ~3.8B active	Efficient server-side reasoning
31B	Dense flagship	31B	Leading dense open model on Arena Text (mid-2026)

This is a deliberately different shape from Kimi K3 or DeepSeek V4: instead of one trillion-parameter MoE flagship, Gemma 4 offers a ladder of models each sized to a realistic hardware tier, from a phone up to a single high-end workstation GPU.

1.3 Dense vs MoE inside one family

E2B, E4B, the 12B model and the 31B flagship are all dense — every parameter runs on every token. Only 26B-A4B is sparse MoE, activating roughly 3.8B of its ~26B total parameters per token. Shipping both design styles in one family lets a team pick dense (simpler serving, no routing) or MoE (more capacity per active-compute dollar) based on their own infrastructure, rather than being locked into one architecture across the whole size range.

---

2. Model configuration

Component	E2B / E4B	12B	26B-A4B	31B
Architecture	Dense	Dense, encoder-free	Sparse MoE	Dense
Parameters	~2.3B / ~4.5B effective	12B	~26B total / ~3.8B active	31B
Vision/audio encoder	Frozen, compact	None — raw patches/frames projected directly	Standard encoder pipeline	Standard encoder pipeline
Attention	Local/global hybrid, sliding window	Local/global hybrid, sliding window	Local/global hybrid, sliding window	Local/global hybrid, sliding window
Quantisation support	QAT (mobile int2/int4 + int8 activations)	QAT, Q4_0	QAT	QAT
Speculative decoding	MTP drafter head	MTP drafter head	MTP drafter head	MTP drafter head
Licence	Apache 2.0	Apache 2.0	Apache 2.0	Apache 2.0

---

3. Architecture

3.1 Local/global hybrid attention

Full self-attention costs grow roughly quadratically with sequence length. Gemma 4 mitigates this the way Gemma 3 did, with a local/global hybrid: most layers use a sliding-window local attention pattern that only looks at nearby tokens, and a minority of layers use full global attention.

The documented ratio is 5:1 local-to-global on most sizes (4:1 on E2B), meaning for every five (or four) sliding-window layers there is one full-attention layer that can retrieve information from anywhere in the sequence. This keeps most of the network cheap while still preserving long-range retrieval where it matters.

3.2 p-RoPE, KV sharing and key-as-value reuse

Global layers are the expensive ones for KV-cache memory. Gemma 4 combines several techniques to shrink that cost further:

• p-RoPE (partial rotary positional embeddings) applied on global layers.
• KV sharing across some layers, so multiple layers reuse the same cached keys/values instead of each maintaining an independent cache.
• Key-as-value reuse in global layers, avoiding a fully separate value cache where the key representation is sufficient.

Combined, Google reports these reduce the global KV-cache footprint by up to roughly 37.5% relative to a non-optimised equivalent — a meaningful difference when the deployment target is a phone or a single consumer GPU rather than a datacentre node.

3.3 Encoder-free 12B: multimodality without a separate vision/audio stack

Most Gemma 4 sizes still use small, frozen vision and audio encoders (a SigLIP-lineage vision pathway, in keeping with Gemma 3's approach) feeding a projector into the language model. The 12B model instead removes those separate encoder stacks entirely:

• Vision: raw image patches (48×48×3 RGB) are projected directly into the LLM's embedding dimension, with explicit 2D coordinate positional embeddings added so the model retains spatial layout information that a convolutional or transformer vision encoder would otherwise encode implicitly.
• Audio: raw audio is segmented into 40 ms frames at 16 kHz and projected directly into the same embedding space; because audio is inherently a temporal sequence, no additional positional encoding is required beyond the model's own sequence handling.
• Shared backbone: text, vision and audio tokens then flow through one transformer stack rather than a separately trained encoder plus adapter.

The practical payoff is memory and latency: no separate encoder weights to load, no cross-attention bridge between a frozen vision tower and the language model, and less memory fragmentation on constrained hardware. Google targets the 12B class at devices with roughly 16GB of suitable VRAM or unified memory, especially once quantised.

3.4 Multi-token prediction (MTP)

Every Gemma 4 size ships an autoregressive MTP drafter head — a lightweight component trained to propose several future tokens at once, which the main model then verifies in a single pass (speculative decoding):

This directly targets local-deployment latency, where decode speed (tokens/second on a single consumer device) matters more than raw throughput at scale.

3.5 Quantisation-aware training (QAT)

Rather than only quantising after training and accepting whatever accuracy loss results, Gemma 4 trains with quantisation noise present, so the network's weights adapt to reduced-precision arithmetic ahead of time:

• Mobile profile: int2/int4 weights with int8 activations, aimed at phones.
• Q4_0 checkpoints: a common llama.cpp-ecosystem 4-bit format, published directly rather than left to community re-quantisation.

QAT checkpoints for E2B/E4B are explicitly sized for phone- and laptop-class memory budgets after quantisation.

---

4. Training and post-training

4.1 Pretraining lineage

Gemma 4 continues the Gemma 3 technical report's (arXiv:2503.19786) broad approach: large-scale multilingual and multimodal pretraining, followed by supervised fine-tuning and preference optimisation. The 12B encoder-free design in particular represents a from-scratch multimodal pretraining choice rather than attaching a pretrained vision/audio encoder after the fact — consistent with the "native multimodal" direction described across this section (compare Kimi K3's MoonViT-V2 and Qwen 3.5's early fusion, both of which still use a dedicated visual encoder rather than raw-patch projection).

4.2 Thinking mode

Gemma 4 supports an optional thinking mode that emits a reasoning trace before the final answer, mirroring the reasoning-budget pattern used across the 2026 field (see Qwen 3.5's thinking/non-thinking switch and Mistral Small 4's reasoning_effort). It is optional rather than always-on, letting latency-sensitive local deployments skip the reasoning trace entirely.

4.3 Capability profile relative to Gemma 3

Google reports meaningful gains over Gemma 3 27B on STEM, coding, multimodal and long-context evaluations:

• E2B roughly matches Gemma 3 27B on several text benchmarks at about one-tenth the parameter count.
• E4B matches or exceeds Gemma 3 27B on vision evaluations.
• 31B and 26B-A4B sit near much larger open MoE models on human-preference rankings, with 31B reported as the leading dense open model on Arena Text as of mid-2026.
• E2B/E4B audio quality improves over Gemma 3n despite a smaller on-disk audio encoder after quantisation.
• The encoder-free 12B still delivers competitive speech transcription and translation despite dropping the dedicated audio encoder.

Read these as directional, benchmark-sourced claims rather than universal guarantees — always validate against your own workload before committing to a size class.

---

5. Deployment

5.1 Choosing a size class by hardware

Hardware	Recommended starting point
Phone / very constrained edge	E2B (mobile QAT int2/int4)
Laptop, ~16GB unified memory or VRAM	E4B or the encoder-free 12B, quantised
Single high-end consumer/workstation GPU (24GB+)	12B or 31B (quantised)
Small server, cost-sensitive reasoning workload	26B-A4B
Single high-memory datacentre GPU	31B at higher precision

5.2 Serving engines

Gemma 4 is supported across the standard local and server-side stack:

Engine	Fit
llama.cpp / Ollama	Q4_0 and GGUF-family quantised checkpoints; CPU and consumer-GPU inference
MLX	Apple Silicon local inference
vLLM / SGLang	Server-side batched inference; 26B-A4B MoE and 31B dense at higher concurrency
Transformers	Reference implementation, fine-tuning, research use

5.3 Example: local inference with Ollama

ollama pull gemma4:12b-q4_0
ollama run gemma4:12b-q4_0 "Summarise this document in three bullet points."

5.4 Example: server-side vLLM for 26B-A4B

docker run --rm \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  google/gemma-4-26b-a4b \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --max-model-len 32768

Start well below the model's maximum supported context and raise it only after benchmarking memory headroom on your specific GPU, following the same discipline recommended for every long-context model in this section.

5.5 Multimodal request example (encoder-free 12B)

from openai import OpenAI
import base64
from pathlib import Path

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

image_bytes = Path("receipt.png").read_bytes()
encoded = base64.b64encode(image_bytes).decode("utf-8")

response = client.chat.completions.create(
    model="google/gemma-4-12b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{encoded}"}},
                {"type": "text", "text": "Extract the total amount and date from this receipt."},
            ],
        }
    ],
)

print(response.choices[0].message.content)

5.6 Fine-tuning considerations

Because Gemma 4 spans dense and MoE designs at very different scales, fine-tuning strategy should match the size class:

Size class	Practical fine-tuning approach
E2B / E4B	Full or LoRA fine-tuning on a single consumer GPU is realistic given the small parameter count
12B (encoder-free)	LoRA/QLoRA on a single workstation GPU; full fine-tuning needs a higher-memory single GPU or small multi-GPU setup
26B-A4B	MoE fine-tuning needs care around expert load balance during training, not just standard LoRA application
31B	Comparable to fine-tuning any 31B dense model; multi-GPU recommended for full fine-tuning

Always re-validate QAT behaviour after fine-tuning — fine-tuning a QAT checkpoint without preserving the quantisation-aware training regime can silently regress the accuracy gains QAT was designed to protect.

5.7 Where Gemma 4 sits relative to other efficiency-focused families

Model	Design philosophy	Smallest practical deployment
Gemma 4	Hardware-tiered ladder (phone → laptop → single GPU)	E2B on a phone
Mistral Small 4	One unified checkpoint, MoE efficiency	~4× H100-class GPUs
GPT-OSS-20b	Single reasoning-focused MoE checkpoint	Single consumer GPU (~16GB, native MXFP4)

Gemma 4 is the only family in this section explicitly engineered down to phone-class hardware; Mistral and GPT-OSS both target single-GPU-to-small-cluster deployment rather than on-device use.

---

6. Common misconceptions

6.1 "Smaller Gemma 4 models are just Gemma 3 with fewer parameters"

No — E2B/E4B use per-layer embeddings and compact frozen encoders as a deliberate on-device design, the 12B model is architecturally distinct (encoder-free, raw-patch/frame projection), and 26B-A4B introduces MoE sparsity that has no equivalent in the dense-only Gemma 3 line. Each size class made independent architectural choices rather than being a uniform scale-down of one design.

6.2 "26B-A4B behaves like a 4B model"

Only in per-token compute. The full ~26B checkpoint must still be resident or shardable, and MoE routing still means different tokens can activate different experts within the same batch — the usual total-vs-active distinction that applies to every MoE model in this section.

6.3 "QAT checkpoints are just quantised after the fact, like community GGUFs"

QAT checkpoints are trained with quantisation noise present during training, so the weights themselves adapt to reduced precision ahead of time. This generally preserves more accuracy at aggressive bit-widths (int2/int4) than applying the same quantisation to a checkpoint that was never trained with that constraint — the two are not interchangeable even when the final bit-width is identical.

6.4 "Encoder-free means Gemma 4 12B has weaker vision than models with a dedicated encoder"

Not according to Google's reported evaluations — the 12B model reportedly delivers competitive results on both vision and speech tasks despite dropping the dedicated encoder stacks used elsewhere in the family. The trade is architectural simplicity and lower memory fragmentation, not necessarily lower quality; validate against your own workload before assuming either direction.

---

7. Troubleshooting local deployment

7.1 Out-of-memory on a phone or laptop

Confirm the correct QAT variant is selected for the target device (mobile int2/int4 for phones, Q4_0 or a similar 4-bit GGUF for laptops) rather than attempting to load a higher-precision checkpoint and quantising client-side. Reduce context length and batch/concurrency before assuming the hardware is simply insufficient.

7.2 Slow decode speed on a single consumer GPU

Confirm the MTP drafter head is actually enabled in the serving configuration — speculative decoding is one of the more direct latency levers available for local deployment, and some serving-engine defaults do not enable it automatically for every checkpoint.

7.3 Vision or audio input silently ignored

For the encoder-free 12B model specifically, verify the serving engine's multimodal input path is using direct patch/frame projection rather than attempting to route through a separate encoder pipeline built for the frozen-encoder sizes (E2B/E4B, 26B-A4B, 31B) — these paths are not interchangeable across the family.

7.4 Unexpectedly long responses despite thinking mode being off

Confirm the chat template's thinking-mode flag is actually being honoured by the serving engine version in use; a template/engine mismatch after an upgrade is the most common cause of a reasoning trace appearing when non-thinking behaviour was expected.

---

8. Licence

Gemma 4 is released under Apache 2.0, the same permissive baseline as Qwen 3.5, Mistral and GPT-OSS. Google additionally publishes Gemma-specific usage guidance and a Responsible GenAI Toolkit; these are governance aids, not additional legal restrictions on top of Apache 2.0. As with every open-weight model, the underlying pretraining corpus is not published in full — Apache 2.0 covers what Google chose to release, not an implicit guarantee about upstream data provenance.

---

9. Engineer reading checklist

• Why local/global hybrid attention with a 5:1 (or 4:1) ratio reduces cost without discarding long-range retrieval
• p-RoPE, KV sharing and key-as-value reuse, and why they specifically target global-layer memory
• Encoder-free multimodality: raw-patch and raw-frame projection vs a dedicated vision/audio encoder
• Quantisation-aware training vs post-hoc quantisation, and why QAT preserves more accuracy at 4-bit and below
• Multi-token prediction as a local-latency tool, not just a datacentre throughput tool
• Matching Gemma 4 size class to actual hardware rather than defaulting to the largest checkpoint

---

References

• ai.google.dev/gemma/docs — Gemma 4 documentation
• Gemma 4 model card (model_card_4)
• Hugging Face: google/gemma-4-* model collection
• GitHub: github.com/google-deepmind/gemma
• Gemma 3 Technical Report — arXiv:2503.19786 (architectural lineage for local/global attention and SigLIP-based vision)
• Google Responsible GenAI Toolkit

Key takeaways

• Gemma 4 is a hardware-tiered family (E2B → E4B → 12B → 26B-A4B → 31B), not a single frontier checkpoint — pick the size class that fits your actual device.
• Local/global hybrid attention plus p-RoPE, KV sharing and key-as-value reuse are what make long-context local inference feasible on limited memory.
• The encoder-free 12B design trades a dedicated vision/audio encoder for direct patch/frame projection — less memory, competitive quality.
• QAT checkpoints (mobile int2/int4, Q4_0) should be preferred over ad hoc post-hoc quantisation for edge deployment.

What to do next

Compare with Qwen 3.5 for a larger-scale hybrid-attention design, browse Open source models, or return to the AI model landscape for local-model selection guidance.