The AI Model Landscape in 2026: Leading Open-Source, Open-Weight and Closed Models by Category

Executive summary

The 2026 model market is multimodal, reasoning-capable and agentic. Choose by openness (fully open, open-weight, closed), workload fit and total cost per successful task — not parameter count alone. Sparse MoE, hybrid attention, million-token context and adjustable reasoning effort define the frontier.

When to use this page

• Briefing stakeholders on open source vs open weight vs closed models.
• Selecting models for enterprise, research, local or agentic workloads.
• Understanding MoE, hybrid attention, multimodality and post-training design.
• Pair with Kimi K3, Model FinOps and Generative AI fundamentals.

Introduction

The artificial-intelligence model market has evolved beyond a simple competition over parameter count. The leading systems of 2026 are increasingly designed as multimodal, reasoning-capable and agentic platforms that can understand documents, images, audio and video; invoke tools; operate software; write and test code; and continue working across long-running tasks.

At the same time, the boundary between open source, open weight and closed source has become increasingly important. Two models may offer similar capabilities, while providing completely different levels of transparency, control, licensing freedom and deployment flexibility.

This article explains:

• What open source, open weight and closed source actually mean.
• How modern foundation models are designed and trained.
• The leading models in each major category as of July 2026.
• The architectural ideas behind models such as Kimi K3, DeepSeek V4, Qwen, GLM, Nemotron, Mistral, Gemma, GPT, Claude and Gemini.
• How to select an appropriate model for enterprise, research, local and agentic workloads.

---

1. Open source, open weight and closed source

1.1 Fully open-source AI

A genuinely open-source AI system provides substantially more than a downloadable checkpoint.

According to the Open Source Initiative, users should have the freedom to:

• Use the system for any purpose.
• Study how it works.
• Modify the system.
• Share the original or modified system.

The preferred materials for modification should include the model parameters, relevant training and inference code, data information, processing methods, architecture, evaluation materials and other components needed to understand how the model was produced.

A strongly open release may therefore provide:

• Final model weights.
• Base and instruction-tuned checkpoints.
• Intermediate checkpoints.
• Model architecture.
• Tokenizer.
• Pretraining and post-training code.
• Data-processing code.
• Training configuration and hyperparameters.
• Training dataset or sufficiently detailed data information.
• Evaluation framework.
• Fine-tuning and inference code.

OLMo is one of the clearest examples of this approach. Ai2 publishes models together with development information intended to make the training process inspectable and reproducible. The Open Source Initiative has also identified OLMo as one of the models that passed its earlier validation exercise.

1.2 Open-weight AI

An open-weight model makes its trained parameters available for download. Developers can normally:

• Run it on their own infrastructure.
• Quantise it.
• Fine-tune or adapt it.
• Integrate it with private systems.
• Inspect its activations and behaviour.
• Deploy it without sending every request to the original provider.

However, the developer may not release:

• The complete training dataset.
• Exact data-selection methodology.
• Full training pipeline.
• Reinforcement-learning environments.
• Human-feedback data.
• Intermediate checkpoints.
• Optimiser states.
• Complete safety-training process.

Most models popularly called “open-source LLMs” are more accurately described as open-weight models.

Examples include:

• Kimi K3.
• DeepSeek V4.
• Qwen open models.
• Llama 4.
• GPT-OSS.
• Many Mistral releases.
• Gemma models.
• GLM open models.

Some use highly permissive licences, while others use custom licences containing usage, attribution or redistribution conditions.

1.3 Closed-source AI

A closed model is accessed through an API, application or managed cloud platform. The provider retains control of the:

• Weights.
• Training code.
• Training data.
• Inference infrastructure.
• Model updates.
• Safety systems.
• Availability and pricing.

Examples include:

• OpenAI GPT-5.6.
• Anthropic Claude Fable 5, Opus 5 and Sonnet 5.
• Google Gemini’s proprietary models.
• Meta Muse Spark.
• Qwen3.7-Max.
• Many specialised image, audio and video services.

A closed model may publish a system card or high-level technical description, but this does not normally provide enough information to reproduce the model.

1.4 Comparison

Dimension	Fully open source	Open weight	Closed source
Download the weights	Yes	Yes	No
Self-hosting	Yes	Usually	No
Fine-tuning	Yes	Usually	Only when provider supports it
Complete training code	Expected	Usually incomplete	No
Training-data information	Detailed or reproducible	Usually limited	Limited
Reproduce the model	Potentially	Generally impossible	Impossible
Inspect internals	Yes	Partly	No
Provider-managed scaling	Optional	Optional	Yes
Data-sovereignty control	High	High	Depends on provider
Operational responsibility	Customer	Customer	Provider
Fastest access to frontier capability	Uncommon	Sometimes	Usually

---

2. How modern AI models are designed

A frontier model is not simply a large neural network. It is the result of several connected design decisions.

2.1 Data pipeline

The training process starts by assembling large quantities of data, potentially including:

• Web documents.
• Books and academic papers.
• Source code.
• Mathematical problems.
• Images and captions.
• Audio and transcriptions.
• Videos.
• Structured tables.
• Tool-use trajectories.
• Human demonstrations.
• Synthetic examples generated by other models.

The raw data must then be processed through:

1. Collection.
2. Language and format identification.
3. Quality filtering.
4. Deduplication.
5. Safety filtering.
6. Personally identifiable information controls.
7. Copyright and licensing controls.
8. Domain balancing.
9. Tokenisation.
10. Packing into training sequences.

Data quality is increasingly important because simply increasing the number of raw tokens produces diminishing returns. Modern developers use classifiers, model-based filtering, execution tests for code and synthetic-data pipelines to improve the useful information contained in the training mixture.

2.2 Tokenisation

Models do not directly process words. Text is divided into tokens, which may represent:

• A complete word.
• Part of a word.
• Punctuation.
• Whitespace.
• A programming-language operator.
• A control instruction.
• A tool call.
• An image or audio representation.

A larger multilingual vocabulary can reduce the number of tokens required to represent some languages. Qwen3.5, for example, expanded its vocabulary to approximately 250,000 tokens and increased support to 201 languages and dialects.

2.3 Dense transformer models

In a dense model, almost all model parameters participate in processing every token.

A simplified transformer block contains:

Dense models are relatively straightforward to train and serve. However, increasing the total parameter count also increases the computation required for every generated token.

Examples include:

• Mistral Medium 3.5, a 128-billion-parameter dense model.
• Gemma 4 12B.
• Many smaller Qwen models.
• Traditional Llama models before the larger MoE releases.

Mistral Medium 3.5 combines instruction following, coding and reasoning within a single 128B dense checkpoint and supports a 256,000-token context.

2.4 Mixture-of-Experts models

A Mixture-of-Experts, or MoE, model contains many feed-forward expert networks but activates only a subset for each token.

Consider a model with:

• 400 billion total parameters.
• 17 billion active parameters per token.

The model can store knowledge and specialised behaviour across the full 400 billion parameters, while using computation closer to a much smaller model for each token.

This distinction creates two parameter measurements:

• Total parameters: all parameters stored in the checkpoint.
• Active parameters: parameters used to process an individual token.

MoE reduces computation, but it does not automatically eliminate storage requirements. The system must still store or distribute the experts unless it uses expert offloading.

Current MoE examples include:

• Kimi K3: 2.8T total, 104B active.
• DeepSeek V4-Pro: 1.6T total, 49B active.
• NVIDIA Nemotron 3 Ultra: 550B total, 55B active.
• Llama 4 Maverick: 400B total, 17B active.
• Qwen3.6-35B-A3B: 35B total, 3B active.
• Mistral Small 4: 119B total, approximately 6B active within its expert layers.

2.5 Attention and the long-context problem

Self-attention allows each token to retrieve information from other tokens in the sequence. Standard full attention becomes expensive as the sequence grows because its computational requirements increase approximately quadratically with context length.

Developers are addressing this with:

• Sliding-window attention.
• Sparse attention.
• Grouped-query attention.
• Multi-latent attention.
• Linear attention.
• Recurrent state-space layers.
• Context compression.
• Retrieval systems.
• External memory.

A one-million-token context is useful only when the model can reliably locate, preserve and reason over relevant information. The maximum accepted input length and the model’s effective retrieval quality are not the same thing.

2.6 Hybrid transformer and recurrent architectures

Several current models combine transformer attention with recurrent or linear-attention components.

Transformers are strong at precise retrieval across a sequence. Recurrent and state-space layers can update a compressed state without repeatedly comparing every token with the full history.

Ai2’s OLMo Hybrid interleaves transformer layers with Gated DeltaNet recurrent layers. Ai2 reports that the hybrid design becomes increasingly advantageous as the context grows, while retaining a fully open development approach.

Other hybrid systems include:

• Qwen3.5 and Qwen3.6.
• Kimi models using Delta Attention.
• NVIDIA Nemotron’s Mamba-attention hybrids.
• Emerging linear-attention research models.

2.7 Multimodal architecture

A multimodal model may process text, images, audio and video.

There are two broad approaches.

Separate encoders

Early or unified fusion

Multimodal information is introduced earlier into the shared network.

Llama 4 uses early fusion for native multimodality. Qwen3.5 also uses early text-vision fusion. Gemma 4 12B goes further by projecting raw image patches and audio frames directly into the language-model input space instead of relying on large, separate encoders.

2.8 Pretraining

During pretraining, the model repeatedly predicts missing or subsequent tokens.

For an autoregressive language model:

Input:  Artificial intelligence is transforming
Target: intelligence is transforming business

The model gradually learns:

• Grammar.
• Facts.
• Programming patterns.
• Mathematical structures.
• Semantic relationships.
• Visual and audio concepts.
• Approximate world models.

Pretraining creates the base model but does not necessarily make it a helpful assistant.

2.9 Post-training

Post-training shapes the base model into a useful product.

It may include:

Supervised fine-tuning

The model learns from carefully constructed examples:

Preference optimisation

The model learns which of several outputs humans or evaluators prefer.

Reinforcement learning

The model receives rewards for producing successful outcomes.

For tasks with verifiable answers, rewards may come from:

• Correct mathematical answers.
• Passing unit tests.
• Successful code execution.
• Correct tool usage.
• Completing a simulated workflow.
• Producing valid structured output.

Distillation

A smaller model learns from a larger teacher model.

Agentic reinforcement learning

The model operates in environments containing:

• Browsers.
• Terminals.
• Code repositories.
• APIs.
• Databases.
• Simulated applications.
• Tool servers.

It is rewarded for completing the overall task rather than merely generating an attractive answer.

2.10 Multi-token prediction and speculative decoding

A conventional language model predicts one next token at a time.

A multi-token prediction component attempts to predict several future tokens. Those predictions can then be verified by the main model, increasing generation speed when multiple predictions are accepted.

NVIDIA Nemotron 3 Ultra includes multi-token prediction layers for native speculative decoding. GLM-5.2 also includes improvements to its MTP component.

2.11 Quantisation

Quantisation stores weights or activations using reduced numerical precision.

Common formats include:

• FP16 or BF16.
• FP8.
• INT8.
• INT4.
• GPTQ.
• AWQ.
• GGUF quantisation.
• MXFP4.
• NVFP4.

Lower precision reduces memory and can increase speed, but aggressive quantisation may reduce accuracy.

Approximate weight memory before runtime overhead is:

Model size	16-bit	8-bit	4-bit
7B	14 GB	7 GB	3.5 GB
12B	24 GB	12 GB	6 GB
20B	40 GB	20 GB	10 GB
35B	70 GB	35 GB	17.5 GB
120B	240 GB	120 GB	60 GB

Actual requirements are higher because the system also needs memory for:

• Key-value cache.
• Activations.
• Runtime kernels.
• Temporary buffers.
• Multimodal encoders.
• Batch processing.

---

3. Leading current closed-source models

3.1 OpenAI GPT-5.6

Deep dive: GPT-5.6 engineering guide. Briefing: GPT-5.6 for AI engineers.

OpenAI released the GPT-5.6 family in July 2026:

• GPT-5.6 Sol: flagship model for difficult professional, coding, scientific and agentic work.
• GPT-5.6 Terra: balance between capability and cost.
• GPT-5.6 Luna: cost-efficient, high-volume model.

The models support adjustable reasoning effort. OpenAI also introduced an Ultra setting for coordinating parallel agent work on particularly demanding tasks. The complete architecture, parameter count, weights and training pipeline remain closed.

Design philosophy

GPT-5.6 is best understood as part of a complete agent platform rather than only as a text generator. Its public capabilities emphasise:

• Reasoning-effort control.
• Coding.
• Computer use.
• Tool invocation.
• Multimodal input.
• Knowledge work.
• Long-running agent execution.
• Coordination of parallel workstreams.

Best suited for

• High-value professional workflows.
• Complex coding and debugging.
• Document and spreadsheet production.
• Computer-use agents.
• Scientific analysis.
• Managed enterprise deployments.

3.2 Anthropic Claude Fable 5, Opus 5 and Sonnet 5

Deep dive: Claude engineering guide. Briefing: Claude 5 for AI engineers.

Anthropic’s 2026 portfolio includes multiple capability tiers.

Claude Fable 5

Fable 5 is Anthropic’s highest generally available capability tier and is designed for frontier coding, knowledge work, vision, science and long-horizon agent execution. Anthropic describes it as a safeguarded version of the same underlying class of model used for Mythos 5.

Claude Opus 5

Released on 24 July 2026, Opus 5 targets serious everyday professional and coding work. Anthropic positions it close to Fable-level capability at lower cost, with particular strength in debugging, root-cause analysis, knowledge work and sustained agentic execution.

Claude Sonnet 5

Sonnet 5 is designed as a scalable execution model for coding, automation, tool use and enterprise workloads where cost and throughput matter.

Claude Mythos 5

Mythos 5 is a restricted model intended for selected cybersecurity, biology and healthcare research partners. It is not a normal general-purpose deployment option.

Anthropic does not publicly release the weights, detailed architecture, parameter count or complete training pipeline for these models.

3.3 Google Gemini

Deep dive: Gemini engineering guide. Briefing: Gemini 3 for AI engineers.

Google’s current portfolio includes different models for frontier, fast, multimodal and media-generation workloads.

The Gemini API lists:

• Gemini 3.6 Flash: latest stable model balancing speed and intelligence.
• Gemini 3.5 Flash: high-capability stable model for agentic and coding tasks.
• Gemini 3.5 Flash-Lite: high-throughput and economical execution model.
• Gemini 3.1 Pro Preview: advanced model for difficult reasoning, agentic and coding workloads.
• Gemini 3.1 Flash Live: real-time voice interactions.
• Gemini 3.1 Flash TTS: speech generation.
• Gemini Omni Flash: conversational video generation and editing.
• Nano Banana models: image generation and editing.

Gemini’s strategic advantage is the breadth of the platform:

• Text.
• Images.
• Audio.
• Video.
• Real-time interaction.
• Tool use.
• Google Search grounding.
• Google Workspace integration.
• Managed agent infrastructure.

3.4 xAI Grok 4.5

Deep dive: Grok engineering guide. Briefing: Grok 4.5 for AI engineers.

Grok 4.5 is xAI’s current flagship for coding, agents and knowledge work. It is API-only and closed-weight. Public materials emphasise large-scale coding reinforcement learning, asynchronous agent rollouts, reasoning-effort controls, search-grounded workloads and long-running software-engineering agents — without disclosing enough architecture or training detail to reproduce the model.

Best suited for

• Coding and debugging agents with tool-heavy harnesses.
• Real-time or search-grounded knowledge work.
• Teams already standardised on xAI’s API or OpenAI-compatible client stacks that also talk to Grok.

3.5 Amazon Nova

Deep dive: Amazon Nova engineering guide. Briefing: Amazon Nova for AI engineers.

Amazon Nova is a closed family available primarily through Amazon Bedrock. The portfolio spans text and multimodal models, reasoning variants (including Nova 2 Lite), speech-to-speech systems (Nova 2 Sonic), embeddings and browser-action agents (Nova Act). Customisation paths now include continued pretraining, supervised fine-tuning and reinforcement fine-tuning with managed deployment through SageMaker.

Best suited for

• AWS-native enterprise deployments that need Guardrails, IAM, provisioned throughput and inference profiles.
• Organisations that prefer Bedrock operational controls over multi-cloud model shopping.
• Browser-use and speech workloads that must stay inside the AWS boundary.

3.6 Meta Muse Spark

Meta introduced Muse Spark in 2026 as a closed model built by Meta Superintelligence Labs. It powers Meta AI and is designed around personal context, planning, application connections and task execution. Meta has indicated that it hopes to open-source future model versions, but Muse Spark itself is not an open-weight Llama release.

3.7 Qwen3.7-Max

Qwen3.7-Max is Qwen’s latest proprietary model for agentic workloads. It should be distinguished from the downloadable Qwen3.5 and Qwen3.6 open-weight families.

---

4. Leading current open and open-weight models

Overview

Model	Openness	Design	Context	Main strength
Kimi K3	Open weight, custom licence	2.8T MoE, KDA, gated MLA	1M	Frontier multimodal agents
DeepSeek V4-Pro	Open weight	1.6T/49B-active MoE	1M	Reasoning and coding
GLM-5.2	Open weights and open tooling	Sparse attention, IndexShare, MTP	1M	Long-horizon agents
Nemotron 3 Ultra	Highly open model stack	550B/55B-active Mamba-attention MoE	1M	Enterprise reasoning
Mistral Medium 3.5	Open weight	128B dense multimodal model	256K	Coding and enterprise agents
Mistral Small 4	Apache 2.0 release	119B/6B-active MoE	256K	Efficient multimodal reasoning
Qwen3.6-35B-A3B	Open weight	35B/3B-active hybrid MoE	128K class	Efficient coding agents
Gemma 4 12B	Apache 2.0 open model	Dense, encoder-free multimodal	Model-specific	Local multimodal AI
GPT-OSS-120B	Apache 2.0 open weight	117B/5.1B-active MoE	128K	Local reasoning and tools
OLMo Hybrid 7B	Fully open	Transformer and Gated DeltaNet	Up to 64K release variants	Reproducible research
Llama 4 Maverick	Open weight, custom licence	400B/17B-active MoE	1M	Broad ecosystem
Llama 4 Scout	Open weight, custom licence	109B/17B-active MoE	10M	Very long inputs

4.1 Kimi K3

Kimi K3 is one of the largest downloadable models released to date.

Its published configuration includes:

• 2.8 trillion total parameters.
• 104 billion activated parameters.
• 93 layers.
• 896 experts.
• 16 routed experts selected per token.
• Two shared experts.
• 69 Kimi Delta Attention layers.
• 24 gated multi-latent attention layers.
• MoonViT-V2 vision encoder.
• One-million-token context.
• MXFP4 weights and MXFP8 activations.
• Native text, image and video-oriented agent workflows.

Kimi Delta Attention

Kimi Delta Attention combines ideas from attention and recurrent state updates. Instead of relying entirely on expensive full attention, it maintains and updates a state that can represent information from the sequence.

This is especially valuable at very long context lengths.

Gated multi-latent attention

Multi-latent attention compresses the information stored for keys and values into a smaller latent representation. This reduces the memory associated with the attention cache.

Gating gives the network additional control over what information is admitted or propagated.

Stable LatentMoE

Kimi K3 increases MoE sparsity to 896 experts while selecting only 16 for a token. Its LatentMoE design aims to improve specialisation and routing stability at this scale.

Practical implications

Kimi K3 is open weight, but it is not a conventional local model. The full checkpoint is extremely large and is intended for:

• Multi-GPU servers.
• High-end inference clusters.
• Specialist hosting providers.
• Enterprise or research infrastructure.

It is particularly relevant for:

• Repository-scale coding.
• Terminal agents.
• Interactive research reports.
• Multimodal analysis.
• CAD and design workflows.
• Long-context document processing.

See the deep dive: Kimi K3: Architecture, Training, Terminology and Local Deployment. Engineer briefing: Kimi K3 open-weight frontier.

4.2 DeepSeek V4

Deep dive: DeepSeek engineering guide. Briefing: DeepSeek MLA, MoE and GRPO.

DeepSeek V4 was released in two principal versions:

• DeepSeek-V4-Pro: 1.6T total and 49B active parameters.
• DeepSeek-V4-Flash: 284B total and 13B active parameters.

Both support one-million-token contexts and thinking or non-thinking operation.

Design focus

DeepSeek’s model family has consistently focused on:

• Sparse MoE scaling.
• Efficient attention and key-value caching.
• Coding.
• Mathematical reasoning.
• Tool use.
• Cost-efficient inference.

The Pro model is intended for maximum reasoning and agent capability, while Flash offers lower latency and cost.

Why DeepSeek is significant

DeepSeek demonstrates that a model can offer:

• Frontier-scale total capacity.
• Much smaller active computation.
• Downloadable weights.
• API access.
• Reasoning and non-reasoning modes.
• Strong compatibility with existing agent frameworks.

4.3 GLM-5.2

Deep dive: GLM-5 engineering guide. Briefing: GLM-5 agentic engineering.

GLM-5.2 is designed specifically for long-horizon work and supports a one-million-token context.

Its notable design elements include:

• Sparse attention.
• IndexShare, which reuses an indexer across groups of sparse-attention layers.
• Improved multi-token prediction.
• Multiple reasoning-effort levels.
• Agentic reinforcement learning.
• On-policy distillation from multiple expert models.
• Support for common self-hosted inference frameworks.

IndexShare reduces the repeated cost of identifying relevant tokens across long contexts. Z.ai reports a substantial reduction in per-token computation at one-million-token length.

GLM-5.2 is especially relevant for:

• Long-running coding.
• Large-repository understanding.
• Tool-heavy agents.
• Extended research workflows.
• Multi-stage document production.

4.4 NVIDIA Nemotron 3 Ultra

Deep dive: Nemotron 3 engineering guide. Briefing: Nemotron 3 hybrid MoE.

Nemotron 3 Ultra contains:

• 550B total parameters.
• 55B active parameters.
• A hybrid Mamba-2 and attention architecture.
• Latent MoE.
• Multi-token prediction.
• NVFP4 pretraining.
• Adjustable reasoning budget.
• Up to one-million-token context.

NVIDIA releases:

• Base checkpoints.
• Post-trained checkpoints.
• Quantised checkpoints.
• Training datasets.
• Post-training data.
• Model recipes.
• Reward-model-related components.

This makes Nemotron particularly important for enterprises and researchers seeking more than a final instruction-tuned checkpoint.

Why Mamba layers matter

Mamba-style state-space layers process sequences through recurrent state updates. Their cost can scale more favourably than full attention over long inputs.

Nemotron combines:

• Attention for precise retrieval.
• Mamba layers for efficient sequence state.
• MoE for capacity.
• Multi-token prediction for faster generation.

4.5 Mistral Medium 3.5

Mistral Medium 3.5 is a 128B dense, open-weight model with a 256K context window.

It unifies:

• Instruction following.
• Reasoning.
• Coding.
• Tool use.
• Vision.
• Long-running agent behaviour.

Mistral states that it can be self-hosted on as few as four appropriate GPUs.

A dense model can be operationally simpler than a large MoE model because it does not require expert routing or expert parallelism. However, all 128B parameters participate in computation.

4.6 Mistral Small 4

Deep dive: Mistral engineering guide. Briefing: Mistral Small 4 unified open model.

Mistral Small 4 contains:

• 119B total parameters.
• Approximately 6B active expert parameters per token.
• 128 experts.
• Four experts active per token.
• Native text and image input.
• 256K context.
• Configurable reasoning effort.
• Apache 2.0 licensing.

It combines capabilities previously distributed across:

• Magistral reasoning models.
• Pixtral vision models.
• Devstral coding models.
• Mistral’s general instruction models.

This makes Small 4 one of the more practical unified open models for organisations that want one checkpoint for chat, vision, reasoning and coding.

4.7 Qwen3.5 and Qwen3.6

Deep dive: Qwen 3.5 engineering guide. Briefing: Qwen 3.5 hybrid MoE.

Qwen3.5 introduced a native multimodal hybrid architecture combining:

• Gated Delta Networks.
• Gated attention.
• Sparse MoE.
• Multi-token prediction.
• Early text-vision fusion.
• FP8 training.
• 201 languages and dialects.

The flagship open Qwen3.5 release contains 397B total parameters and 17B active parameters.

Qwen3.6-35B-A3B reduces this to:

• 35B total parameters.
• 3B active parameters.
• Multimodal input.
• Thinking and non-thinking modes.
• Strong agentic coding capability.

It is designed as a highly efficient coding and tool-use model that can be self-hosted more practically than frontier-scale MoE systems.

4.8 Gemma 4

Deep dive: Gemma 4 engineering guide. Briefing: Gemma 4 local multimodal.

Gemma 4 is Google DeepMind’s open-weight multimodal family for local, edge and efficient server deployment. Released under Apache 2.0 (technical report dated June 2026), it spans dense and Mixture-of-Experts architectures from roughly 2.3B to 31B parameters, with native text, image and audio support and an optional thinking mode that emits a reasoning trace before the final answer.

The published suite includes:

• E2B / E4B: effective 2.3B and 4.5B dense models (with per-layer embeddings) plus compact frozen vision and audio encoders for on-device work.
• 12B: dense model trained from scratch with a unified encoder-free multimodal design.
• 26B-A4B: MoE with about 26B total and 3.8B activated parameters.
• 31B: dense flagship; the leading dense open model on Arena Text as of mid-2026, competitive with much larger MoE systems.

Encoder-free 12B design

Most Gemma 4 sizes still use frozen vision and audio encoders. The 12B model replaces them with lightweight projections into the shared language-model space:

• Raw image patches (48×48×3 RGB) are projected directly into the LLM embedding dimension, with 2D coordinate positional embeddings for spatial structure.
• Raw audio is segmented into 40 ms frames at 16 kHz and projected into the same space; as a temporal sequence it needs no extra positional encoding.
• Text, vision and audio then share one transformer backbone.

This removes separate encoder stacks, cuts memory fragmentation and lowers latency for multimodal local inference. The 12B class is aimed at devices with roughly 16GB of suitable VRAM or unified memory, especially when quantised.

Efficiency features across the family

Beyond the 12B architecture, Gemma 4 emphasises deployment cost:

• Long-context memory: a 5:1 local sliding-window to global attention pattern (4:1 on E2B), p-RoPE on global layers, KV sharing and key-as-value reuse in global layers reduce the global KV-cache footprint by up to about 37.5%.
• Multi-token prediction (MTP): an autoregressive MTP drafter head supports speculative decoding for faster local generation.
• Quantisation-aware training (QAT): mobile (int2/int4 weights with int8 activations) and Q4_0 checkpoints shrink weight memory with limited quality loss; quantized E2B/E4B footprints are designed for phones and laptops.

Capability profile

Relative to Gemma 3 27B, Gemma 4 improves strongly on STEM, coding, multimodal and long-context benchmarks. E2B roughly matches Gemma 3 27B on several text tasks at about one-tenth the parameter count; E4B matches or exceeds Gemma 3 27B on vision evals; 31B and 26B-A4B sit near much larger open MoE models in human preference rankings. Audio quality on E2B/E4B improves over Gemma 3n despite a much smaller on-disk audio encoder after quantisation, and the encoder-free 12B still delivers competitive speech transcription and translation.

Gemma 4 is particularly attractive for:

• Offline assistants.
• Private document processing.
• Local audio and visual understanding.
• Mobile and desktop agents.
• IoT and robotics prototypes.
• On-prem reasoning where Apache 2.0 licensing and downloadable weights matter more than frontier closed APIs.

4.9 GPT-OSS

Deep dive: gpt-oss engineering guide. Briefing: gpt-oss Apache reasoning.

OpenAI’s GPT-OSS family includes:

• GPT-OSS-120B: 117B total, approximately 5.1B active parameters.
• GPT-OSS-20B: 21B total, approximately 3.6B active parameters.

Both are MoE reasoning models released under Apache 2.0 and support up to 128K context. GPT-OSS-120B is intended to fit within approximately 80GB of memory, while GPT-OSS-20B can operate within approximately 16GB under its native quantisation configuration.

Their architecture includes:

• Alternating dense and locally banded sparse attention.
• Grouped multi-query attention.
• Rotary positional embeddings.
• MXFP4 quantised weights.
• Configurable reasoning effort.
• Structured tool calling.

They are appropriate for:

• Private reasoning systems.
• On-premises agents.
• Local tool use.
• Custom safety classifiers.
• Organisations that prefer Apache-licensed weights.

4.10 OLMo Hybrid

OLMo Hybrid is a 7B fully open model combining:

• Transformer attention.
• Gated DeltaNet recurrent layers.
• Open training information.
• Open model weights.
• Open development artefacts.
• Long-context adaptation.

Ai2 developed it through controlled comparisons with OLMo 3, making it valuable for understanding whether hybrid architectures genuinely improve data and inference efficiency.

It may not match the raw capability of trillion-parameter frontier systems, but it is substantially more useful for:

• Reproducible research.
• Architecture experiments.
• Training studies.
• Data research.
• Interpretability.
• University projects.

4.11 Llama 4

Deep dive: Llama 4 engineering guide. Briefing: Llama 4 MoE and licence.

Llama 4 remains an important open-weight ecosystem even though Meta’s newest proprietary model line has moved to Muse.

Llama 4 Scout

• 109B total parameters.
• 17B active.
• 16 experts.
• 10-million-token context.
• Native text and image input.

Llama 4 Maverick

• 400B total parameters.
• 17B active.
• 128 experts.
• One-million-token context.
• Native text and image input.

Both use MoE and early-fusion multimodality. They are distributed under the custom Llama Community Licence rather than a standard OSI-approved software licence.

Llama remains relevant because of its:

• Large tooling ecosystem.
• Cloud availability.
• Fine-tuning community.
• Quantisation support.
• Extensive deployment documentation.

---

5. Leading models by workload category

5.1 General-purpose frontier intelligence

Closed models

Leading current options include:

• GPT-5.6 Sol.
• Claude Fable 5.
• Claude Opus 5.
• Gemini 3.1 Pro Preview.
• Qwen3.7-Max.

These models are suitable when maximum capability is more important than self-hosting.

Open or open-weight models

Leading options include:

• Kimi K3.
• DeepSeek V4-Pro.
• GLM-5.2.
• Nemotron 3 Ultra.
• Qwen3.5-397B-A17B.
• Mistral Medium 3.5.

These provide greater infrastructure control but require considerably more operational expertise.

5.2 Reasoning

Reasoning models spend additional inference computation before producing their final answer.

Strong current options include:

Closed

• GPT-5.6 Sol at high or maximum reasoning.
• Claude Fable 5.
• Claude Opus 5.
• Gemini Pro-class models.

Open weight

• DeepSeek V4-Pro.
• Kimi K3.
• GLM-5.2.
• Nemotron 3 Ultra.
• Qwen3.5.
• GPT-OSS-120B.
• Mistral Small 4.

The best model depends on whether the reasoning task is:

• Mathematical.
• Scientific.
• Coding-oriented.
• Document-based.
• Tool-assisted.
• Visual.
• Long horizon.

5.3 Coding and software-engineering agents

A coding agent requires more than code completion. It must be able to:

1. Explore a repository.
2. Understand dependencies.
3. Form a plan.
4. Edit multiple files.
5. Run commands.
6. Interpret test failures.
7. Debug its own work.
8. Preserve project conventions.
9. Verify the final implementation.

Leading closed models include:

• GPT-5.6 Sol.
• Claude Fable 5.
• Claude Opus 5.
• Claude Sonnet 5.
• Gemini 3.5 Flash and Gemini 3.1 Pro.

Leading open-weight options include:

• Kimi K3.
• DeepSeek V4-Pro.
• GLM-5.2.
• Mistral Medium 3.5.
• Qwen3.6-35B-A3B.
• Nemotron 3 Ultra.

For coding agents, the agent harness can be as important as the underlying model. The harness controls:

• Repository search.
• Terminal access.
• Context management.
• Test execution.
• Permission boundaries.
• Subagents.
• Checkpoints.
• Retry logic.
• Human approval.

5.4 Multimodal understanding

For analysing images, diagrams, screenshots, documents, audio or video, strong closed options include:

• GPT-5.6.
• Claude Fable 5 and Opus 5.
• Gemini models.

Strong downloadable options include:

• Kimi K3.
• Qwen3.5.
• Gemma 4.
• Mistral Small 4.
• Llama 4.
• Qwen3-Omni.

Qwen3-Omni uses separate Thinker and Talker components, allowing it to process mixed text, image, audio and video inputs and produce text or audio outputs.

5.5 Small and local models

For local deployment, the practical choices are different from the frontier leaderboard.

Good candidates include:

• Gemma 4 12B.
• GPT-OSS-20B.
• OLMo Hybrid 7B.
• Smaller Qwen models.
• Qwen3.6-35B-A3B on higher-memory systems.
• Quantised Mistral models.
• Smaller Llama variants.

Important considerations include:

• Available RAM or VRAM.
• Prompt length.
• Required output speed.
• Quantisation quality.
• Whether image or audio input is needed.
• Number of concurrent users.
• Tool-use requirements.

A 3B-active MoE model is not necessarily stored like a 3B dense model. Active parameters determine computation, while total parameters still influence checkpoint and memory requirements.

5.6 Research and reproducibility

For architecture, training-data or interpretability research, the strongest choice is not necessarily the model with the highest benchmark score.

Important candidates include:

• OLMo and OLMo Hybrid.
• NVIDIA Nemotron.
• EleutherAI model families.
• Selected BigScience and BigCode projects.
• Qwen-Scope interpretability artefacts.

Qwen-Scope publishes sparse autoencoders trained on hidden layers of multiple Qwen models, allowing researchers to study more interpretable internal features.

5.7 Multilingual models

Strong multilingual options include:

• Qwen3.5 and Qwen3.6.
• Gemini.
• GPT.
• Claude.
• Aya.
• Mistral.
• Gemma 4.

Cohere Labs’ Aya programme focuses specifically on multilingual access, including regional Tiny Aya models and broader research releases.

---

6. Embedding and RAG models

A generative model produces text. An embedding model produces a numerical vector representing the meaning of text, images or other information.

Embeddings support:

• Semantic search.
• Retrieval-augmented generation.
• Document clustering.
• Recommendation systems.
• Duplicate detection.
• Classification.
• Reranking.

Qwen3 Embedding

Qwen3 Embedding provides embedding and reranking models based on the Qwen3 foundation model. The models were released under Apache 2.0 and use multilingual synthetic and weakly supervised data for retrieval-related tasks.

Bi-encoder retrieval

A bi-encoder separately converts the query and document into vectors:

It is fast enough to search millions of documents.

Cross-encoder reranking

A reranker processes the query and document together:

It is slower but usually more accurate. A common enterprise RAG pipeline therefore uses:

1. Embedding retrieval to find 50 candidates.
2. Reranking to select the best 5–10.
3. A generative model to create the answer.
4. Citations linking the answer to the retrieved sources.

---

7. Speech and voice models

7.1 Closed voice systems

GPT-Live

GPT-Live is OpenAI’s full-duplex voice architecture. Full duplex means the system can listen and speak simultaneously rather than forcing strict alternating turns. It can maintain the conversation while delegating harder work to a frontier reasoning model.

Gemini Live and TTS

Google offers:

• Gemini 3.1 Flash Live for real-time conversations.
• Gemini 3.5 Live Translate.
• Gemini 3.1 Flash TTS.
• Pro-oriented TTS models for higher-quality structured audio.

7.2 Open and open-weight audio models

Whisper

Whisper is an MIT-licensed encoder-decoder transformer for:

• Speech recognition.
• Language identification.
• Speech translation.
• Multilingual transcription.
• Voice activity-related token prediction.

Whisper Turbo is an optimised version of Large-v3 that improves speed with limited degradation.

Voxtral TTS

Mistral’s Voxtral TTS is a 4B model containing:

• A 3.4B transformer decoder.
• A 390M flow-matching acoustic transformer.
• A 300M neural audio codec.

The language backbone predicts semantic information while the flow-matching component generates acoustic representations used by the codec to synthesise speech.

---

8. Image-generation models

8.1 Closed models

GPT Image 2

GPT Image 2 is OpenAI’s current flagship image-generation and editing model. It focuses on:

• Precise instruction following.
• Text rendering.
• Image editing.
• Layout.
• Multilingual content.
• Visual reasoning.
• Production asset creation.

Google Nano Banana

Google’s current image family includes:

• Nano Banana 2.
• Nano Banana 2 Lite.
• Nano Banana Pro.

These models support native image generation and editing through the Gemini ecosystem.

8.2 Downloadable and open image models

Qwen-Image-3.0

Qwen released Qwen-Image-3.0 in July 2026 as the latest generation of its image model family, focusing on realistic content, text rendering and knowledge-rich visual generation.

Diffusion architecture

Most dedicated image models use some form of diffusion or flow-based generation:

1. Start with noise or a compressed latent.
2. Condition on the text prompt and optional reference images.
3. Iteratively remove noise or move the latent towards the target distribution.
4. Decode the final latent into an image.

Newer multimodal language models may integrate visual generation more tightly with language reasoning, enabling conversational editing and stronger understanding of complex instructions.

---

9. Video-generation models

9.1 Closed video models

Google Veo 3.1

Veo 3.1 supports:

• Text-to-video.
• Image-to-video.
• Video with audio.
• Scene extension.
• Creative controls.
• Improved physical realism.
• Prompt adherence.

Sora 2

Sora 2 combines video and audio generation and focuses on:

• Physical consistency.
• Controllable multi-shot sequences.
• Sound effects.
• Dialogue.
• Image-guided generation.
• Video editing and extension through the API.

Gemini Omni Flash

Gemini Omni Flash provides conversational video generation and editing, allowing users to produce and refine video through natural-language interaction.

9.2 Open and open-weight video models

Wan 2.2

Wan 2.2 provides open video-generation models for:

• Text-to-video.
• Image-to-video.
• Combined text and image guidance.
• Character animation.
• Speech-driven video.

Its 5B hybrid text/image-to-video model supports 720p at 24 frames per second and is designed to run on high-end consumer GPUs such as an RTX 4090.

HunyuanVideo 1.5

HunyuanVideo 1.5 is an 8.3B model designed for consumer-grade deployment.

Its architecture includes:

• Diffusion Transformer components.
• Selective and sliding-tile attention.
• Bilingual text encoding.
• Progressive pretraining.
• Video super-resolution.
• Distilled low-step inference variants.

---

10. How open and closed models differ technically

The internal neural-network ideas may be similar. Both open and closed models can use:

• Transformers.
• MoE.
• Reinforcement learning.
• Multimodal fusion.
• Tool training.
• Sparse attention.
• Quantisation.
• Synthetic data.

The main difference is not necessarily the architecture. It is what the provider discloses and what the adopter controls.

Closed-model design

Advantages:

• Easy deployment.
• Automatic scaling.
• Continuous upgrades.
• Managed safety.
• No GPU administration.
• Immediate access to frontier models.

Limitations:

• Limited internal visibility.
• Vendor dependency.
• Variable model behaviour after updates.
• API and data-residency considerations.
• Less control over quantisation and inference.
• Limited reproducibility.

Open-weight design

Advantages:

• Private deployment.
• Data sovereignty.
• Custom fine-tuning.
• Model inspection.
• Quantisation.
• Provider independence.
• Potentially predictable unit economics at scale.

Limitations:

• GPU costs.
• Capacity planning.
• Security patching.
• Model evaluation.
• Safety engineering.
• Observability.
• Upgrade management.
• Specialist operational skills.

Fully open research design

This provides the greatest research value because teams can investigate not just the final model, but how the model came into existence.

---

11. How to select a model

Choose a closed frontier model when

• The organisation needs maximum capability immediately.
• The workload is irregular and does not justify dedicated GPUs.
• Managed scaling is important.
• The team lacks model-serving expertise.
• The provider’s security and data terms satisfy the use case.
• Rapid product development matters more than infrastructure control.

Strong candidates:

• GPT-5.6.
• Claude Fable 5 or Opus 5.
• Gemini’s current stable and Pro models.

Choose an open-weight frontier model when

• Data must remain within a controlled environment.
• The organisation needs model customisation.
• Inference volume is high enough to justify dedicated infrastructure.
• The team needs predictable model versions.
• The system must function without dependence on an external API.
• The organisation can support model operations.

Strong candidates:

• Kimi K3.
• DeepSeek V4.
• GLM-5.2.
• Nemotron 3 Ultra.
• Mistral Medium 3.5.
• Qwen3.5.

Choose an efficient local model when

• The application must run offline.
• User privacy is central.
• Latency must be very low.
• The workload is narrow or repetitive.
• The device has limited memory.
• A smaller model can be combined with RAG and tools.

Strong candidates:

• Gemma 4 12B.
• GPT-OSS-20B.
• OLMo Hybrid 7B.
• Smaller Qwen and Mistral models.

Choose a fully open model when

• Reproducibility is required.
• The purpose is academic research.
• The team needs to study training behaviour.
• The organisation wants to audit data and architecture.
• Interpretability and experimentation matter more than frontier benchmark leadership.

Strong candidates:

• OLMo.
• OLMo Hybrid.
• Selected Nemotron releases.
• Open research projects from Ai2, EleutherAI, BigScience and BigCode.

---

12. Enterprise evaluation framework

An enterprise should not select a model using one public leaderboard.

A proper evaluation should measure:

Capability

• Task success rate.
• Reasoning accuracy.
• Coding quality.
• Retrieval performance.
• Multimodal understanding.
• Tool-use reliability.

Operational performance

• Time to first token.
• Tokens per second.
• End-to-end task time.
• Maximum concurrency.
• GPU utilisation.
• Failure and retry rates.

Quality and safety

• Hallucination rate.
• Citation correctness.
• Prompt-injection resistance.
• Sensitive-data leakage.
• Bias.
• Unsafe tool actions.
• Human-escalation behaviour.

Commercial factors

• Cost per successful task.
• Infrastructure cost.
• API cost.
• Engineering support.
• Licensing.
• Exit and migration costs.

Governance

• Model-card quality.
• Data provenance information.
• Evaluation documentation.
• Audit logs.
• Version control.
• Regulatory support.
• Incident-management process.

The most meaningful metric is often not cost per token, but:

Total cost per successfully completed, verified business task.

A cheap model that repeatedly fails, requires retries or needs extensive human correction may be more expensive than a higher-priced model that completes the work correctly.

---

13. Major architectural trends

13.1 Increasing MoE sparsity

Models are adding more total experts while activating fewer experts per token. This increases capacity without proportionally increasing computation.

Examples include:

• Kimi K3.
• DeepSeek V4.
• Nemotron 3.
• Qwen3.5 and Qwen3.6.
• Mistral Small 4.

13.2 Hybrid attention and recurrent models

Full attention is increasingly being combined with:

• Delta networks.
• Mamba layers.
• Linear attention.
• Sparse retrieval.

This is intended to make million-token contexts more practical.

13.3 Reasoning as an adjustable resource

Models increasingly expose a reasoning-effort parameter:

• None or fast.
• Low.
• Medium.
• High.
• Maximum.

The same model can therefore serve both quick conversational requests and difficult analytical tasks.

13.4 Native multimodality

Instead of attaching separate image and audio systems after language training, new models increasingly train multiple modalities together.

This improves:

• Cross-modal reasoning.
• Visual tool use.
• Video understanding.
• Spoken interaction.
• GUI agents.
• Document analysis.

13.5 Models becoming agent foundations

Training is moving from isolated question answering to complete workflows.

Models are increasingly evaluated on whether they can:

• Use a terminal.
• Browse.
• Write and execute code.
• Call APIs.
• Create documents.
• Operate user interfaces.
• Coordinate subagents.
• Verify their own outputs.
• Continue working for hours or days.

13.6 Model and system co-design

The best AI product is no longer determined solely by its foundation model.

Performance depends on the complete system:

A smaller model inside a well-designed system can outperform a larger model inside a poorly designed one.

---

Conclusion

The 2026 AI landscape contains three distinct forms of availability.

Closed frontier models such as GPT-5.6, Claude Fable 5, Claude Opus 5 and Gemini provide the fastest access to highly capable managed intelligence.

Open-weight models such as Kimi K3, DeepSeek V4, GLM-5.2, Qwen, Mistral, Gemma, GPT-OSS and Llama provide greater control, self-hosting and customisation, but vary significantly in licence terms and training transparency.

Fully open models such as OLMo provide the strongest foundation for reproducible research because they expose not only a final checkpoint but substantially more of the model-development process.

The most important architectural developments are:

• Sparse Mixture-of-Experts.
• Hybrid attention and recurrent layers.
• Million-token contexts.
• Native multimodality.
• Adjustable reasoning effort.
• Multi-token prediction.
• Agentic reinforcement learning.
• Quantisation-aware training.
• Unified models capable of reasoning, coding, perception and tool use.

There is therefore no universally best model. The correct choice depends on the organisation’s required capability, privacy, latency, scale, infrastructure, governance, licensing and total cost per successful task.

Key takeaways

• Open-weight is not fully open source; licences and training transparency still vary widely.
• MoE capacity (total parameters) and active compute are different measurements — plan storage and GPU memory accordingly.
• Evaluate on cost per successful verified task, not cost per token alone.
• The agent harness and surrounding system often matter as much as the foundation model.

What to do next

Continue with the Kimi K3 deep dive, browse Open source models, or plan capacity with Model FinOps.