Large Language Model Architectures Explained: Mathematics, Programs, Analogies, and Data Flow

An LLM architecture defines how a model converts language into numbers, moves information between tokens, stores short-term contextual memory, transforms representations through neural layers, predicts or generates tokens, and scales parameters and computation.

Most modern language models belong to one or more of these families:

• Recurrent architectures: RNN, LSTM, GRU and RWKV
• Encoder-only Transformers: BERT-style models
• Decoder-only Transformers: GPT, Llama and Mistral-style models
• Encoder–decoder Transformers: T5 and BART-style models
• Sparse Mixture-of-Experts models
• Local, sparse and linear-attention models
• Retention and delta-rule architectures
• State-space models such as Mamba
• Convolutional sequence models such as Hyena
• Hybrid attention–recurrent architectures
• Diffusion language models
• Multimodal language models
• Retrieval-augmented language-model systems

1. The fundamental language-model problem​

Suppose a tokenized document is:

A standard autoregressive language model represents the probability of the sequence as:

For the sentence:

The customer cancelled the order.

The model learns:

then:

and eventually:

The training loss is usually cross-entropy, or negative log-likelihood:

where:

• (x_{<t}) means all tokens before position (t)
• (\theta) represents all model parameters
• (P_\theta) is the probability distribution produced by the model

A lower loss means the model assigned a higher probability to the correct next token.

1.1 Perplexity​

Language-model quality is also frequently expressed using perplexity:

Perplexity can be interpreted approximately as the model’s effective uncertainty among possible next tokens. Lower perplexity is normally better, although it does not directly measure factuality, reasoning, safety or usefulness.

2. The universal LLM pipeline​

Despite their differences, many LLMs follow this high-level structure:

For a decoder-only Transformer, the concrete architecture usually resembles:

3. Tokenization​

Neural networks cannot directly process words. Text must first be converted into tokens.

A tokenizer might convert:

"unbelievable architecture"

into:

["un", "believ", "able", " architecture"]

Each token is mapped to an integer:

"un"            → 451
"believ"        → 9281
"able"          → 731
" architecture" → 15220

The result becomes:

3.1 Why use subword tokens?​

A word-level vocabulary would struggle with rare words, misspellings, new names, technical terminology, different word endings and multilingual text. Subword tokenization allows models to compose unfamiliar words from smaller units.

Common approaches include:

• Byte Pair Encoding (BPE)
• WordPiece
• Unigram tokenization
• Byte-level tokenization
• SentencePiece

Tokenization is not merely preprocessing. It affects context-window usage, multilingual performance, mathematical reasoning, code representation, inference cost and vocabulary-head size.

4. Embeddings​

Assume vocabulary size (V=50{,}000) and hidden dimension (d=4096).

The embedding matrix is:

For token ID (i), the embedding is:

Therefore every token becomes a vector of 4,096 floating-point numbers. For a sequence of (T) tokens:

4.1 Real-world analogy​

Think of each token as a person carrying a 4,096-field identity card. The fields do not have simple labels such as “noun” or “positive.” Meaning is distributed across the vector. Different dimensions jointly encode syntactic, semantic and contextual properties.

4.2 Embedding parameter count​

For (V=50{,}000) and (d=4096):

The embedding table alone contains about 205 million parameters.

Many models tie the input embedding matrix to the output vocabulary projection:

This reduces the number of independent parameters.

5. Early neural language-model architectures​

Before Transformers, language models were dominated by recurrent and convolutional architectures.

5.1 Feed-forward neural language models​

A fixed number of previous tokens is used to predict the next token. For a context window of three:

The embeddings are concatenated:

Then transformed:

Limitation. The context length is fixed. Information ten or one hundred tokens earlier cannot be considered unless the input window is expanded dramatically.

6. Recurrent Neural Networks​

An RNN processes tokens sequentially:

where:

• (x_t) is the current token representation
• (h_{t-1}) is the previous hidden state
• (h_t) is the new hidden state

The output distribution is:

6.1 Architecture​

The same RNN parameters are reused at every position.

6.2 Real-world analogy​

Imagine one employee reading a report one word at a time and maintaining a single notebook. After every word, the employee updates the notebook. The notebook is the hidden state. The problem is that the notebook has limited space—as the report becomes longer, earlier details may be overwritten or weakened.

6.3 Vanishing gradients​

During backpropagation, gradients repeatedly multiply by recurrent matrices:

If the magnitude of these products is below one, gradients shrink exponentially. If it is above one, gradients may explode. This makes learning very long dependencies difficult.

7. LSTM architecture​

Long Short-Term Memory networks introduce a persistent cell state and several gates.

7.1 Forget gate​

The forget gate decides what information should be removed from memory.

7.2 Input gate and candidate memory​

7.3 Cell-state update​

7.4 Output gate and hidden state​

Here, (\odot) means element-wise multiplication.

7.5 Real-world analogy​

An LSTM is like an administrator managing a filing cabinet:

• Forget gate: which old files should be destroyed?
• Input gate: which new files should be stored?
• Cell state: the filing cabinet
• Output gate: which files should be shown to the next department?

LSTMs preserve information better than simple RNNs, but training remains sequential. Token (t+1) cannot be fully processed before token (t).

8. GRU architecture​

A Gated Recurrent Unit simplifies the LSTM.

8.1 Update and reset gates​

8.2 Candidate and final state​

The update gate controls how much old information is retained versus replaced.

9. Sequence-to-sequence RNNs​

Early translation systems used an encoder RNN and decoder RNN.

The original weakness was that the entire source sentence had to be compressed into one fixed-size context vector. Attention was introduced to allow the decoder to inspect all encoder states.

For decoder state (s_t) and encoder states (h_j):

The decoder receives the weighted context (c_t). This idea developed into Transformer attention.

10. The Transformer​

The Transformer removed recurrence and used attention as the main sequence-processing mechanism. The original architecture contained an encoder stack and decoder stack, with multi-head attention, feed-forward networks, residual connections and normalization.

10.1 Core idea​

Each token can directly inspect other relevant tokens. Instead of passing information through:

the model can form direct relationships:

11. Query, key and value​

Given:

the model calculates:

where:

Each token therefore produces:

• A query: what information am I looking for?
• A key: what kind of information do I contain?
• A value: what information should I provide?

11.1 Real-world analogy​

Imagine a business networking event. Every participant carries a question card (query), a nameplate describing their expertise (key), and a folder containing their actual knowledge (value). A participant compares their question card with everyone’s nameplate. Participants with relevant expertise receive more attention. The participant then combines information from their folders.

12. Scaled dot-product attention​

The similarity between queries and keys is:

For token (i) attending to token (j):

The score is scaled and normalized:

Therefore:

12.1 Why divide by (\sqrt{d_h})?​

If query and key components have variance around one, their dot product has variance proportional to (d_h). Large values can make softmax extremely sharp, producing small gradients. Dividing by (\sqrt{d_h}) stabilizes the score scale.

13. A numerical attention example​

Suppose a query is:

and three keys are:

The raw similarities are:

After softmax, the approximate weights are:

If the values are:

then:

The output combines information from all three tokens, weighted by relevance.

14. Causal attention​

A generative language model must not inspect future tokens during training. A causal mask is added:

Attention becomes:

The attention matrix has a triangular structure:

Token 1: ✓ ✗ ✗ ✗
Token 2: ✓ ✓ ✗ ✗
Token 3: ✓ ✓ ✓ ✗
Token 4: ✓ ✓ ✓ ✓

This allows all training positions to be calculated in parallel while preserving next-token prediction.

15. Multi-head attention​

One attention operation may learn one type of relationship. Multi-head attention creates several parallel attention mechanisms.

For head (i):

The heads are concatenated and projected:

Different heads may specialize in local syntax, subject–verb relationships, entity references, previous definitions, formatting patterns or long-range dependencies. It is not guaranteed that every head has a simple human-interpretable role.

15.1 Parameter count​

For standard multi-head attention:

Ignoring biases:

For (d=4096):

That is about 67 million parameters per attention module.

16. PyTorch self-attention implementation​

import math

import torch
import torch.nn as nn
import torch.nn.functional as F


class CausalSelfAttention(nn.Module):
    def __init__(
        self,
        d_model: int,
        n_heads: int,
        max_seq_len: int,
        dropout: float = 0.0,
    ) -> None:
        super().__init__()

        if d_model % n_heads != 0:
            raise ValueError("d_model must be divisible by n_heads")

        self.d_model = d_model
        self.n_heads = n_heads
        self.head_dim = d_model // n_heads

        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)
        self.output_projection = nn.Linear(d_model, d_model, bias=False)
        self.dropout = nn.Dropout(dropout)

        causal_mask = torch.tril(
            torch.ones(max_seq_len, max_seq_len, dtype=torch.bool)
        )
        self.register_buffer(
            "causal_mask",
            causal_mask.view(1, 1, max_seq_len, max_seq_len),
            persistent=False,
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        batch_size, seq_len, d_model = x.shape

        if d_model != self.d_model:
            raise ValueError("Unexpected hidden dimension")

        q, k, v = self.qkv(x).chunk(3, dim=-1)

        q = q.view(batch_size, seq_len, self.n_heads, self.head_dim).transpose(1, 2)
        k = k.view(batch_size, seq_len, self.n_heads, self.head_dim).transpose(1, 2)
        v = v.view(batch_size, seq_len, self.n_heads, self.head_dim).transpose(1, 2)

        scores = (q @ k.transpose(-2, -1)) / math.sqrt(self.head_dim)
        mask = self.causal_mask[:, :, :seq_len, :seq_len]
        scores = scores.masked_fill(~mask, float("-inf"))

        attention_weights = self.dropout(F.softmax(scores, dim=-1))
        output = attention_weights @ v
        output = output.transpose(1, 2).contiguous().view(batch_size, seq_len, d_model)
        return self.output_projection(output)

Modern libraries often use fused kernels such as scaled dot-product attention rather than explicitly creating the full score matrix.

17. Attention complexity​

For sequence length (T) and hidden dimension (d), computing (QK^\top) requires approximately (O(T^2 d)) operations. The attention matrix requires (O(T^2)) memory per head before kernel-level optimizations.

Doubling sequence length approximately quadruples the size of the attention matrix:

This quadratic dependence is the main motivation for local attention, sparse attention, linear attention, retention and state-space models.

18. Feed-forward networks​

After attention, each token independently passes through a feed-forward network. The original Transformer used:

where:

Usually (d_{\mathrm{ff}}>d). A traditional choice is (d_{\mathrm{ff}}\approx 4d).

18.1 Real-world analogy​

Attention is the communication stage: tokens exchange information. The feed-forward network is the private thinking stage: each token processes the information it has gathered.

18.2 Standard FFN parameter count​

If (d=4096) and (d_{\mathrm{ff}}=16384):

The feed-forward network can therefore contain significantly more parameters than attention.

19. SwiGLU​

Many modern decoder models use gated feed-forward networks. A common SwiGLU formulation is:

where:

SwiGLU uses three major matrices (W_g), (W_u) and (W_d). Its parameter count is approximately (3dd_{\mathrm{ff}}). In practice, the intermediate width may be adjusted so the total parameter budget remains comparable to a conventional FFN.

class SwiGLU(nn.Module):
    def __init__(self, d_model: int, d_ff: int) -> None:
        super().__init__()
        self.gate = nn.Linear(d_model, d_ff, bias=False)
        self.up = nn.Linear(d_model, d_ff, bias=False)
        self.down = nn.Linear(d_ff, d_model, bias=False)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.down(F.silu(self.gate(x)) * self.up(x))

20. Residual connections​

A residual connection adds the input to the transformed output:

Instead of forcing a layer to learn a complete new representation, the layer learns a correction to the existing representation.

Analogy. A consultant edits an existing business document rather than rewriting the entire document at every stage. Residuals help preserve information, stabilize optimization, improve gradient flow and train deeper networks.

21. LayerNorm and RMSNorm​

21.1 Layer normalization​

For vector (x\in\mathbb{R}^d):

21.2 RMSNorm​

RMSNorm does not subtract the mean:

Llama-style models popularized a decoder design using pre-normalization, RMSNorm, rotary position embeddings and SwiGLU-like feed-forward blocks.

class RMSNorm(nn.Module):
    def __init__(self, d_model: int, eps: float = 1e-6) -> None:
        super().__init__()
        self.weight = nn.Parameter(torch.ones(d_model))
        self.eps = eps

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        rms = x.pow(2).mean(dim=-1, keepdim=True)
        normalized = x * torch.rsqrt(rms + self.eps)
        return self.weight * normalized

22. Pre-norm and post-norm Transformers​

22.1 Original post-norm design​

22.2 Modern pre-norm design​

Pre-norm generally provides a more direct residual gradient path, which is useful for deep decoder models.

23. Position information​

Attention itself does not inherently know token order. Without positional information, the model would have difficulty distinguishing:

Dog bites person

from:

Person bites dog

23.1 Learned absolute embeddings​

Each position has a learned vector (p_1,p_2,\ldots,p_T). The initial representation is:

This is simple, but the position table usually has a fixed trained length.

23.2 Sinusoidal positions​

The original Transformer used:

Different dimensions oscillate at different frequencies.

23.3 Rotary Position Embeddings (RoPE)​

RoPE rotates pairs of query and key dimensions based on position. For a two-dimensional pair:

The rotated query and key are (q_m'=R_m q) and (k_n'=R_n k). Their dot product becomes:

The interaction therefore depends on the relative distance (n-m), even though each vector is rotated using its absolute position.

23.4 ALiBi​

Attention with Linear Biases adds a distance penalty directly to the attention score:

where (m_h) is a head-specific slope. More distant previous tokens receive a larger negative bias. ALiBi was designed to support length extrapolation without learned positional embeddings.

24. The three principal Transformer families​

24.1 Encoder-only architecture​

Examples include BERT-style models.

Every token may attend to tokens on both sides:

Token 1: ✓ ✓ ✓ ✓
Token 2: ✓ ✓ ✓ ✓
Token 3: ✓ ✓ ✓ ✓
Token 4: ✓ ✓ ✓ ✓

Masked language modelling​

Given The customer [MASK] the order., the model predicts cancelled. The objective is:

where (M) is the set of masked positions.

Strengths. Classification, sentiment analysis, NER, semantic search, reranking, extractive QA and document embeddings.

Weakness. Not naturally designed for long-form left-to-right generation.

24.2 Decoder-only architecture​

Examples include GPT, Llama and many modern chat models.

The model learns:

GPT-3 demonstrated that sufficiently scaled autoregressive models could perform many tasks through zero-shot, one-shot and few-shot prompting without gradient updates for each task.

Why decoder-only models dominate general-purpose generation. The training objective exactly matches generation: train by predicting the next token; infer by predicting the next token. The same architecture supports conversation, code completion, summarization, reasoning-style generation, tool-call generation, structured output, translation and content generation.

24.3 Encoder–decoder architecture​

Examples include the original Transformer, T5 and BART-style models.

The encoder uses bidirectional self-attention. The decoder contains:

1. Causal self-attention over generated tokens
2. Cross-attention over encoder states
3. A feed-forward network

Cross-attention​

Let decoder representations be (D) and encoder representations be (E):

Analogy. The encoder is a research team that reads and organizes a source document. The decoder is a report writer. At every writing step, the decoder consults the research team through cross-attention.

Suitable tasks. Translation, summarization, text correction, structured transformation, question answering and converting documents into another textual format.

25. A complete miniature GPT architecture​

The following program implements the central components of a decoder-only Transformer.

from dataclasses import dataclass

import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class GPTConfig:
    vocab_size: int = 32_000
    max_seq_len: int = 1_024
    d_model: int = 512
    n_heads: int = 8
    n_layers: int = 8
    d_ff: int = 1_376
    dropout: float = 0.0


class TransformerBlock(nn.Module):
    def __init__(self, config: GPTConfig) -> None:
        super().__init__()
        self.attention_norm = RMSNorm(config.d_model)
        self.attention = CausalSelfAttention(
            d_model=config.d_model,
            n_heads=config.n_heads,
            max_seq_len=config.max_seq_len,
            dropout=config.dropout,
        )
        self.ffn_norm = RMSNorm(config.d_model)
        self.ffn = SwiGLU(config.d_model, config.d_ff)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = x + self.attention(self.attention_norm(x))
        x = x + self.ffn(self.ffn_norm(x))
        return x


class TinyGPT(nn.Module):
    def __init__(self, config: GPTConfig) -> None:
        super().__init__()
        self.config = config
        self.token_embedding = nn.Embedding(config.vocab_size, config.d_model)
        # Learned positions for readability; production models often use RoPE.
        self.position_embedding = nn.Embedding(config.max_seq_len, config.d_model)
        self.blocks = nn.ModuleList(
            [TransformerBlock(config) for _ in range(config.n_layers)]
        )
        self.final_norm = RMSNorm(config.d_model)
        self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
        self.lm_head.weight = self.token_embedding.weight  # weight tying

    def forward(
        self,
        token_ids: torch.Tensor,
        targets: torch.Tensor | None = None,
    ) -> tuple[torch.Tensor, torch.Tensor | None]:
        batch_size, seq_len = token_ids.shape
        if seq_len > self.config.max_seq_len:
            raise ValueError("Sequence exceeds max_seq_len")

        positions = torch.arange(seq_len, device=token_ids.device)
        x = self.token_embedding(token_ids) + self.position_embedding(positions)[None, :, :]
        for block in self.blocks:
            x = block(x)
        logits = self.lm_head(self.final_norm(x))

        loss = None
        if targets is not None:
            loss = F.cross_entropy(
                logits.reshape(-1, logits.size(-1)),
                targets.reshape(-1),
            )
        return logits, loss

    @torch.no_grad()
    def generate(
        self,
        token_ids: torch.Tensor,
        max_new_tokens: int,
        temperature: float = 1.0,
        top_k: int | None = None,
    ) -> torch.Tensor:
        self.eval()
        for _ in range(max_new_tokens):
            context = token_ids[:, -self.config.max_seq_len :]
            logits, _ = self(context)
            next_logits = logits[:, -1, :] / max(temperature, 1e-5)
            if top_k is not None:
                values, _ = torch.topk(next_logits, min(top_k, next_logits.size(-1)))
                threshold = values[:, [-1]]
                next_logits = next_logits.masked_fill(
                    next_logits < threshold, float("-inf")
                )
            probabilities = F.softmax(next_logits, dim=-1)
            next_token = torch.multinomial(probabilities, num_samples=1)
            token_ids = torch.cat([token_ids, next_token], dim=1)
        return token_ids

For training, input and target sequences are shifted:

tokens = batch_of_token_ids
inputs = tokens[:, :-1]
targets = tokens[:, 1:]
logits, loss = model(inputs, targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()

26. Multi-head, multi-query and grouped-query attention​

During autoregressive generation, previously calculated keys and values are stored in a KV cache.

26.1 Multi-head attention (MHA)​

Each query head has its own key and value head:

This offers maximum flexibility but creates a large KV cache.

26.2 Multi-query attention (MQA)​

All query heads share one key head and one value head:

This substantially reduces KV-cache memory and memory bandwidth.

26.3 Grouped-query attention (GQA)​

Several query heads share each key-value head:

GQA is a compromise between the quality flexibility of multi-head attention and the inference efficiency of multi-query attention.

Example. Suppose (H_Q=32). With MHA, (H_{KV}=32). With MQA, (H_{KV}=1). With GQA, (H_{KV}=8)—each KV head serves four query heads.

27. KV-cache mathematics​

For (L) layers, batch size (B), cached tokens (T), (H_{KV}) key-value heads, head dimension (d_h) and (b) bytes per element, the KV-cache size is approximately:

The factor 2 represents keys and values.

27.1 Example​

Assume (L=32), (B=1), (T=32{,}768), (H_{KV}=8), (d_h=128) and (b=2):

With 32 KV heads rather than 8, the cache would be approximately four times larger.

28. Sliding-window attention​

Full causal attention allows token (i) to inspect all earlier tokens. Sliding-window attention restricts token (i) to the previous (W) tokens:

Complexity becomes approximately (O(TWd)) instead of (O(T^2d)) when (W\ll T).

Mistral 7B combined grouped-query attention with sliding-window attention to reduce inference and long-sequence costs.

Analogy. Full attention is an employee rereading the complete project archive before writing every sentence. Sliding-window attention is an employee reviewing only the most recent section. Stacked layers allow information to move beyond one local window, but precise access to very distant tokens can still be weaker than full attention.

29. Sparse attention​

Sparse-attention architectures allow only selected connections:

Local attention:        neighbouring tokens
Strided attention:      fixed intervals
Global tokens:          special tokens attend to the whole sequence
Block sparse attention: tokens attend within selected blocks
Routing attention:      tokens grouped by learned or approximate similarity

If each token attends to (k) positions, cost is (O(Tkd)) rather than (O(T^2d)). The challenge is choosing a sparse pattern that does not remove important relationships.

30. Mixture-of-Experts architecture​

A dense Transformer sends every token through the same feed-forward network. A Mixture-of-Experts model contains multiple expert feed-forward networks and selects only a few for each token.

30.1 Router mathematics​

For token representation (x):

where (E_e) is expert (e).

30.2 Total versus active parameters​

Suppose there are eight experts, each containing one billion parameters, and two experts are activated. The model has roughly 8 billion expert parameters but activates approximately 2 billion expert parameters per token, plus shared attention and embedding parameters. This allows a large parameter capacity without activating every parameter for every token.

30.3 Router program​

class SparseMoE(nn.Module):
    def __init__(
        self,
        d_model: int,
        d_ff: int,
        n_experts: int,
        top_k: int = 2,
    ) -> None:
        super().__init__()
        if not 1 <= top_k <= n_experts:
            raise ValueError("Invalid top_k")
        self.n_experts = n_experts
        self.top_k = top_k
        self.router = nn.Linear(d_model, n_experts, bias=False)
        self.experts = nn.ModuleList([SwiGLU(d_model, d_ff) for _ in range(n_experts)])

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        original_shape = x.shape
        flat_x = x.reshape(-1, original_shape[-1])
        router_probs = F.softmax(self.router(flat_x), dim=-1)
        top_probs, top_indices = torch.topk(router_probs, self.top_k, dim=-1)
        top_probs = top_probs / top_probs.sum(dim=-1, keepdim=True)

        output = torch.zeros_like(flat_x)
        for expert_id, expert in enumerate(self.experts):
            token_positions, selected_slots = torch.where(top_indices == expert_id)
            if token_positions.numel() == 0:
                continue
            expert_output = expert(flat_x[token_positions])
            weights = top_probs[token_positions, selected_slots].unsqueeze(-1)
            output.index_add_(0, token_positions, expert_output * weights)
        return output.view(original_shape)

A production MoE implementation must additionally handle token capacity per expert, load balancing, distributed all-to-all communication, expert parallelism, dropped or rerouted tokens, router precision and auxiliary routing losses.

30.4 Load-balancing loss​

Without balancing, one expert may receive most tokens. A simplified balancing term encourages router probability and actual token allocation to remain distributed:

where (f_e) is the fraction of tokens assigned to expert (e) and (p_e) is the average router probability for expert (e).

31. Linear attention​

Softmax attention calculates (\operatorname{softmax}(QK^\top)V). This requires pairwise query–key interactions.

Linear attention replaces the softmax kernel with a feature map (\phi):

For causal attention:

Define recurrent states:

This avoids explicitly creating a (T\times T) attention matrix and reveals a recurrent interpretation of attention.

Analogy. Softmax attention searches the entire archive separately for every query. Linear attention maintains a running summary of the archive. Each new query reads the summary instead of comparing itself with every previous document.

Trade-off. A finite-size summary may compress or mix different key-value associations. This can weaken exact retrieval compared with full attention.

32. DeltaNet​

Simple linear attention adds every key-value association to memory:

Over time, similar keys may collide. DeltaNet treats memory as an associative mapping and corrects the prediction for the current key.

First predict (\hat{v}_t=S_{t-1}k_t), calculate the error (e_t=v_t-\hat{v}_t), then update:

The output for query (q_t) is:

This resembles one online gradient-descent update to make the memory map (k_t) to (v_t). Hardware-efficient chunkwise algorithms make DeltaNet-style models more practical to train.

Analogy. A conventional linear-memory system adds another note to a crowded whiteboard. DeltaNet first checks what the whiteboard currently says for that subject, erases the incorrect portion, and writes the corrected value.

33. Gated DeltaNet​

Delta updates provide targeted correction, but a model also needs to forget stale information. A simplified gated update is:

where (0\le g_t\le 1). The gate controls global or head-level memory decay, while the delta term corrects a specific key-value association.

Gated DeltaNet combines adaptive forgetting with delta-rule updates and has also been studied in hybrid designs containing local attention or state-space layers.

34. Kimi Delta Attention and Kimi Linear​

Kimi Delta Attention extends gated delta-style memory using finer-grained, channel-wise control over forgetting rather than only a coarse head-level gate. A simplified conceptual form is:

Each memory feature can decay at a different rate. The delta update then writes corrected information into the transformed state.

Kimi Linear uses a hybrid structure with three KDA-style layers for each periodic full-attention layer. The full-attention layers restore direct global token interaction, while KDA layers provide efficient recurrent memory. The reported architecture targets lower KV-cache requirements and higher long-context decoding throughput while retaining periodic global attention.

Analogy. A conventional forget gate has one retention policy for an entire department. Channel-wise gating gives every filing category its own retention policy—financial records may be retained strongly, temporary formatting details forgotten quickly, entity names decay differently, and current task instructions remain highly active.

See also From GPT-2 to Kimi Delta Attention.

35. Retentive Networks​

Retention combines attention-like parallel training with recurrent inference. A simplified retention operation is:

where (D) is a causal decay matrix:

The equivalent recurrent state can be written as:

Depending on orientation, the outer product may instead be written (k_t v_t^\top); the central idea is that the state stores a decayed key-value summary.

35.1 Three execution modes​

• Parallel — useful for training: (Y=(QK^\top\odot D)V)
• Recurrent — useful for one-token-at-a-time decoding: (S_t=\gamma S_{t-1}+\text{new association})
• Chunkwise recurrent — tokens are processed in parallel within each chunk, while chunks communicate through recurrent states

36. State-space models​

State-space models represent sequence processing through a latent dynamical system.

36.1 Continuous-time form​

where (x(t)) is input, (h(t)) is hidden state, (y(t)) is output, (A) is the state-transition matrix, (B) maps input to state, (C) maps state to output and (D) is a direct input-to-output mapping.

36.2 Discrete form​

After discretization:

For a step size (\Delta):

Interpretation. The matrix (A) determines how memory evolves. Eigenvalues with slow decay preserve information; rapidly decaying components forget quickly; oscillatory components can represent periodic patterns.

37. Convolutional interpretation of state-space models​

Repeatedly expand the recurrence:

Ignoring the initial state:

Define a convolution kernel (K_i=C\bar{A}^i\bar{B}). Then (y=K*x).

This means a time-invariant state-space model can be calculated recurrently, as a convolution, or through parallel scan algorithms.

38. Mamba architecture​

Traditional state-space parameters are independent of the current token. That limits content-dependent behaviour. Mamba makes important state-space parameters functions of the input:

The recurrence becomes approximately:

Because (\Delta_t), (B_t) and (C_t) depend on the input, the model can selectively retain, ignore, write and read information.

38.1 Simplified Mamba block​

A schematic formulation is:

Analogy. A Transformer keeps a searchable copy of all earlier token representations. Mamba maintains a compressed evolving state. Each new token decides what should enter memory, what should be ignored, which part of memory should influence the output, and how quickly previous information should decay.

Strengths. Linear scaling with sequence length; constant-size recurrent state during streaming inference; no conventional Transformer KV cache; efficient processing of long sequences.

Limitation. Compressed state may make exact arbitrary retrieval more difficult than direct full attention—hence hybrid architectures combining state-space layers and attention.

39. RWKV​

RWKV combines Transformer-like training characteristics with RNN-like recurrent inference. Its main blocks include time mixing, channel mixing, receptance gates and weighted key-value accumulation.

A simplified memory form is:

Actual RWKV implementations use more sophisticated, numerically stable recurrences and token-shift operations. RWKV was designed so that training can be parallelized while inference operates with a constant-size recurrent state.

Analogy. Instead of keeping every meeting transcript, RWKV maintains a continuously updated weighted summary. Important recent or highly weighted information contributes more strongly.

40. Hyena architecture​

Hyena replaces attention with long convolutions and data-dependent gating. A schematic Hyena-style operation can be represented as:

where (*) is a long convolution, (h_1,h_2) are implicitly parameterized long filters and (g_1,g_2) are data-dependent gates.

Hyena was proposed as a subquadratic alternative to attention using long implicit convolutions interleaved with gating.

Analogy. Attention asks which exact earlier tokens should I inspect? Hyena asks which learned long-range filters should I apply to the entire sequence, and how should the current content gate those filters?

41. Hybrid architectures​

No single sequence mechanism is ideal for every requirement.

Full attention offers precise token-to-token retrieval, strong in-context learning and direct access to distant information. Recurrent, linear or state-space mechanisms offer lower memory consumption, efficient streaming, better scaling with sequence length and smaller or constant recurrent state.

A hybrid model may use:

Layer 1: Linear attention
Layer 2: Linear attention
Layer 3: State-space layer
Layer 4: Full attention
Layer 5: Linear attention
Layer 6: Linear attention
Layer 7: State-space layer
Layer 8: Full attention

The efficient layers compress and propagate information. Periodic full-attention layers allow exact global interaction—similar to an organisation where most work is handled through departmental summaries, while periodic cross-company meetings allow direct communication between everyone.

42. Diffusion language models​

Autoregressive models generate strictly left to right:

A masked diffusion language model begins with a partially or completely masked sequence and repeatedly predicts and reveals tokens:

Step 1: [MASK] customer [MASK] order
Step 2: The customer [MASK] order
Step 3: The customer cancelled the order

LLaDA demonstrated an LLM trained using a forward masking process and a reverse denoising process parameterized by a bidirectional Transformer.

42.1 Forward masking process​

Let (x_0) be the original sequence. At noise level (t), tokens are independently masked with probability related to (t):

At high noise levels, most tokens become masks.

42.2 Reverse process​

The model learns (p_\theta(x_{t-\Delta}\mid x_t)). It predicts clean tokens from a noisier sequence. A simplified training loss is:

where (M_t) contains masked positions.

Benefits. Many token positions may be updated in parallel; generation can revise tokens rather than committing permanently from left to right; bidirectional context is naturally available during denoising.

Trade-offs. Each denoising step processes the sequence again. A model may need multiple denoising passes, and a Transformer denoiser may still have quadratic attention cost per pass.

The relevant comparison is:

where (N) may be much smaller than (T), but each pass is more expensive than one cached autoregressive step.

43. Multimodal language-model architectures​

A multimodal model must transform images, audio or video into representations compatible with the language model.

43.1 Projector architecture​

LLaVA connected a vision encoder to an LLM and trained the system using visual instruction data.

Analogy. The vision encoder speaks “visual language.” The LLM speaks “text-vector language.” The projector acts as an interpreter.

43.2 Querying Transformer architecture​

BLIP-2 uses learned query tokens that cross-attend to frozen image features.

Let learned queries be (Q_L\in\mathbb{R}^{m\times d}). They attend to image features (Z_v):

Benefit. A large number of image patches can be compressed into a smaller number of learned visual tokens.

43.3 Gated cross-attention​

Flamingo-style models insert cross-attention layers into a pretrained language model:

A layer may perform:

The learnable gate (g) controls how strongly visual information changes the language representation.

43.4 Native multimodal token models​

Another approach converts different modalities into token-like units:

These are processed by one shared or partially shared Transformer. The challenge is balancing different token rates, information density, positional structures, modality-specific encoders, cross-modal alignment and context-window consumption.

44. Retrieval-Augmented Generation​

RAG is primarily a system architecture, not a replacement for the internal Transformer architecture.

The probabilistic formulation can be written:

where (x) is the question, (z) is a retrieved document, (P_\eta(z\mid x)) is the retriever and (P_\theta(y\mid x,z)) is the generator.

RAG combines parametric memory in model weights with non-parametric memory in an external index.

Analogy. A closed-book LLM answers from memory. A RAG system allows the LLM to visit a library, retrieve relevant documents and answer using those documents.

44.1 Production RAG architecture​

Retriever options include dense embedding retrieval, BM25, hybrid retrieval, cross-encoder reranking, graph traversal, metadata filtering, multi-query retrieval and parent–child retrieval.

45. LoRA and adaptation architecture​

LoRA is not a new backbone. It is a parameter-efficient way of adapting an existing architecture.

For a pretrained weight (W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}):

with:

The original (W) is frozen, while (A) and (B) are trained. LoRA reduces trainable parameters by representing the update in a low-rank subspace.

45.1 Example​

For (W\in\mathbb{R}^{4096\times 4096}), full fine-tuning updates:

parameters. With rank (r=16):

trainable parameters—approximately 128 times fewer for that matrix.

46. Parameter-count estimation for a decoder LLM​

For a dense decoder with vocabulary (V), hidden size (d), layers (L) and FFN width (d_{\mathrm{ff}}), a rough estimate is:

For a standard FFN, (P_{\mathrm{FFN}}\approx 2dd_{\mathrm{ff}}). For SwiGLU, (P_{\mathrm{FFN}}\approx 3dd_{\mathrm{ff}}). Therefore:

Normalization parameters and biases are relatively small and can be added separately.

46.1 Example​

Assume (V=50{,}000), (d=4096), (L=32) and (d_{\mathrm{ff}}=11008).

• Embeddings: (Vd=204.8) million
• Attention per layer: (4d^2\approx 67.1) million
• SwiGLU per layer: (3dd_{\mathrm{ff}}\approx 135.3) million
• Total blocks: (32(67.1+135.3)\approx 6.48) billion

Adding embeddings and small components gives a model in approximately the 6–7 billion parameter range.

47. Training architecture versus inference architecture​

A model’s mathematical architecture and its serving architecture are related but different.

47.1 Training​

Training may use data, tensor, pipeline, sequence and expert parallelism; fully sharded data parallelism; activation checkpointing; and mixed precision.

47.2 Inference​

Prefill. The entire prompt is processed. For Transformers, prompt attention is expensive because many prompt tokens interact.

Decode. One new token is processed at a time. The model reuses cached keys and values rather than recalculating the entire prompt.

48. Softmax and token selection​

The final hidden state is projected to vocabulary logits:

Softmax produces:

48.1 Temperature​

• (\tau<1): sharper, more deterministic
• (\tau>1): flatter, more diverse
• (\tau\to 0): approaches greedy selection

Top-k retains only the (k) highest-probability tokens. Top-p retains the smallest token set whose cumulative probability exceeds (p).

Sampling is not part of the learned neural architecture, but it strongly affects visible model behaviour.

49. Architecture comparison​

The complexity terms are high-level approximations. Actual latency depends heavily on hardware, kernel implementation, batch size, hidden width, communication, memory bandwidth and cache management.

50. Mapping well-known models to architecture families​

51. Which architecture should be used?​

Document classification​

Use an encoder-only model when the entire document is available, bidirectional context is valuable, output is a class or compact representation, and generation is unnecessary.

General conversational AI​

Use a decoder-only model when the system must generate free-form responses, in-context learning is important, tool calls or structured outputs are needed, and a large pretrained ecosystem is desirable.

Translation or controlled transformation​

Use an encoder–decoder model when input and output have clearly separate roles, the full source should be encoded before generation, and conditional generation quality matters.

Extremely large parameter capacity​

Use MoE when the infrastructure can support expert parallelism, model capacity matters more than deployment simplicity, and only a subset of parameters should activate per token.

Streaming and very long sequences​

Consider Mamba, RWKV, retention or hybrid architectures when constant-state streaming is important, KV-cache growth is a major constraint, contexts are extremely long, and exact retrieval can be supplemented by attention or external retrieval.

Enterprise knowledge applications​

Use a decoder model combined with RAG when knowledge changes frequently, source attribution is required, private documents must remain outside model weights, and access control and document-level governance matter.

52. Worked enterprise architecture example​

Consider a banking customer-service assistant.

52.1 System architecture​

52.2 Inside the LLM​

52.3 Information flow example​

Prompt:

Customer:
Why was my international transfer rejected?

Retrieved policy:
Transfers may be rejected when beneficiary details do not
match verification records.

Tool result:
Reason code: BENEFICIARY_NAME_MISMATCH

Attention may connect "rejected" ↔ "BENEFICIARY_NAME_MISMATCH" ↔ "beneficiary details do not match". The feed-forward network transforms the combined representation into a useful response pattern.

The surrounding architecture should still enforce unsupported-claim checks, access control, PII handling, tool-response validation, human escalation rules, audit logs, prompt-injection controls and source traceability.

The neural architecture produces language. The production system architecture determines whether that language is safe, grounded and operationally reliable.

53. The central architectural trade-off​

Nearly every modern LLM architecture is balancing four requirements:

Full attention provides high retrieval precision but expensive context scaling. Recurrent and state-space systems provide efficient memory but compress the past. MoE increases parameter capacity but creates routing and distributed-systems complexity. Diffusion allows parallel revision but requires repeated denoising. Multimodal models gain additional capabilities but must align different representation spaces. RAG adds external knowledge but introduces retrieval, ranking and evidence-management failure modes.

There is therefore no universally best architecture. The best design depends on context length, latency target, hardware, batch size, required factuality, streaming requirements, memory budget, modality, deployment environment, fine-tuning strategy and exact-retrieval requirements.

54. Final mental model​

The entire field can be understood through one question:

How does the current token access and transform information from the past?

The modern trend is not simply toward one winner. It is toward hybrid architectures that combine:

• Direct attention for precise retrieval
• Recurrent or state-space memory for efficiency
• Sparse experts for parameter capacity
• External retrieval for current knowledge
• Multimodal encoders for perception
• Tool execution for real-world actions