Building a GPT-style large language model yourself is the fastest way to stop treating foundation models as black boxes. This playbook walks through the full path used in educational GPT implementations: prepare text, implement attention, assemble a decoder-only stack, pretrain (or load weights), then fine-tune for classification or instruction following.

The conceptual sequence mirrors production LLM development at smaller scale. You can run the educational path on a laptop; frontier training still needs datacenter compute. Use this article as an engineering map, then implement with LLMs-from-scratch and Sebastian Raschka’s Build a Large Language Model (From Scratch) (Manning).

0. What you are actually building​

An LLM in this context is a decoder-only transformer trained primarily with next-token prediction, then optionally adapted:

Why next-token prediction works: language is sequential. Predicting the next token forces the model to compress syntax, semantics, and world regularities into its weights. Emergent abilities (summarize, translate, classify with few examples) appear as side effects of that objective at sufficient scale—not as separately programmed modules.

Why build small first: a ~124M-parameter GPT-2-style model on a short public-domain corpus teaches every moving part. Loading open GPT-2 weights into your architecture then lets you practice fine-tuning without paying for pretraining.

1. Mental model: three stages, nine steps​

STAGE 1 — Implement the machine
  1) Data preparation & sampling
  2) Attention mechanism
  3) LLM (GPT) architecture

STAGE 2 — Create a foundation model
  4) Pretraining on unlabeled text
  5) Training loop
  6) Model evaluation
  7) Load pretrained weights (practical shortcut)

STAGE 3 — Specialize
  8) Fine-tuning for classification
  9) Fine-tuning for instruction following

Everything below expands those nine steps into engineering practice: algorithms, shapes, losses, negative cases, and checks.

Part I — Stage 1: Data, attention, architecture

2. Stage 1.1 — Working with text data​

Deep networks cannot consume raw strings. You need a deterministic pipeline from characters to batched tensors.

2.1 Embeddings: the contract with the network​

An embedding maps discrete symbols to continuous vectors. For GPT-like LMs:

• Token embeddings — one vector per vocabulary ID (lookup table / nn.Embedding).
• Positional embeddings — inject order (GPT uses learned absolute positions, not only relative schemes).
• Input to the stack — usually token_emb + pos_emb (same dimensionality).

Dimensionality is a capacity/cost trade-off (educational GPT-2 small often uses 768; larger variants use 1024–1600+).

Negative case: using the same embedding for audio and text, or assuming Word2Vec static vectors are “good enough” for modern LLMs. LLMs typically learn embeddings end-to-end and also produce contextualized representations inside the stack.

2.2 Tokenization​

Goal: split text into a finite vocabulary of tokens (words, subwords, or bytes).

Educational progression often used in from-scratch curricula:

1. Regex / whitespace split for intuition.
2. Special handling for punctuation.
3. Production path: byte pair encoding (BPE) as in GPT-2.

BPE intuition: start from characters; iteratively merge frequent pairs into subwords. Unknown words decompose into known pieces—no mandatory <|unk|> for every OOV string.

Practical tooling: OpenAI’s tiktoken (Rust-backed) exposes GPT-2 encodings. Always pin versions in experiments.

Special tokens (conceptual):

Negative cases:

• Lowercasing everything and destroying proper-noun signal.
• Removing whitespace when training on indentation-sensitive code.
• Building a tiny vocab from one short story, then encoding “Hello” → KeyError (motivation for BPE / larger corpora).

2.3 From tokens to IDs to batches​

1. Encode full corpus → list of token IDs.
2. Choose context length T (educational: 256; GPT-2 native: 1024).
3. Sliding window with stride:
   • Inputs x[i:i+T]
   • Targets y[i+1:i+T+1] (next-token shift)
4. Batch into tensors shaped [batch, T].

Stride design:

• stride = 1 → dense overlapping windows (more samples, more redundancy / overfitting risk on tiny data).
• stride = T → non-overlapping chunks (efficient coverage).

PyTorch pattern: custom Dataset + DataLoader with drop_last=True during training to avoid undersized batches that spike loss.

Negative cases:

• Forgetting the +1 shift (targets identical to inputs → nonsense learning signal).
• Context length larger than model positional table.
• Mixing train and validation windows from the same contiguous region without a clean split.

2.4 Positional information​

Self-attention is permutation-equivariant without positions. Absolute learned positional embeddings (GPT style) add a position-index vector to each token embedding.

Checks:

• Position IDs must be 0 … seq_len-1 on the correct device.
• Truncate sequences that exceed supported context.
• At inference, growing context must stay ≤ context_length.

3. Stage 1.2 — Attention mechanisms​

Attention is how the model selectively mixes information across the sequence.

3.1 Why attention replaced classic RNN bottlenecks​

Older encoder–decoder RNNs compressed an entire source sentence into a single hidden state before decoding—fragile for long inputs. Attention lets the decoder (and later, self-attention layers) look back at all positions with learned weights.

Transformers push this further: self-attention lets every position attend to every other position in the same sequence (with causal restrictions for GPT).

3.2 Simplified self-attention (intuition)​

Given embeddings X of shape [n_tokens, d]:

1. Compute raw scores via pairwise similarity (often dot products): score_ij = x_i · x_j.
2. Softmax over keys for each query → attention weights summing to 1.
3. Context vector = weighted sum of value vectors (in the toy case, values = inputs).

Why softmax: stable positive weights, interpretable as a distribution; better gradient behaviour than naive sum-normalization on extreme scores.

3.3 Trainable scaled dot-product attention​

Introduce projections:

• Q = X W_Q
• K = X W_K
• V = X W_V

Scores: Q Kᵀ / √d_k, then softmax → weights, then weights @ V.

Why scale by √d_k: large d_k inflates dot products → softmax saturates → vanishing gradients. Scaling keeps logits in a healthier range.

Query / key / value metaphor:

• Query — “what am I looking for?”
• Key — “what do I contain as an index?”
• Value — “what content do I contribute if matched?”

3.4 Causal (masked) attention​

For left-to-right LMs, position t must not attend to t+1 … T.

Implementation pattern:

1. Build upper-triangular mask of -inf.
2. Add to scores before softmax.
3. Softmax turns -inf into ~0 probability.

Efficient trick: mask with -inf then softmax once—avoid mask-after-softmax then renormalize (though both can be made correct if done carefully).

Information leakage check: any nonzero attention weight above the diagonal in causal mode is a bug.

3.5 Dropout on attention weights​

During training, randomly zero some attention weights (then rescale). Typical rates: 0.1–0.2 for GPT-like training; high rates (0.5) are for illustration. Disable in eval().

3.6 Multi-head attention​

Run several attention “heads” in parallel (different projections), then concatenate and project:

MultiHead(X) = Concat(head_1, …, head_h) W_O

Efficient implementation: one large QKV projection, reshape to [batch, heads, tokens, head_dim], batched matmuls, merge heads.

Shape discipline (memorize this):

Negative cases:

• d_model not divisible by num_heads.
• Wrong transpose order → silent shape bugs and garbage attention.
• Forgetting the output projection when matching reference architectures/weights.

4. Stage 1.3 — Coding a GPT model​

4.1 Building blocks beyond attention​

LayerNorm: normalize last dimension to mean ~0 / variance ~1, then learnable scale and shift. Stabilizes deep stacks. Prefer Pre-LN (norm before sublayers) for modern GPT-like training dynamics.

GELU: smooth nonlinearity used in GPT-2 FFNs (often with a tanh approximation for speed).

Feed-forward network: expand (d → 4d), GELU, project back (4d → d). Same input/output width enables stacking without shape surgery.

Residual (shortcut) connections: x ← x + Sublayer(x). Critical against vanishing gradients in deep nets.

4.2 Transformer block (GPT-style sketch)​

x = x + Dropout(Attention(LayerNorm(x)))
x = x + Dropout(FFN(LayerNorm(x)))

Repeat n_layers times (e.g. 12 for GPT-2 small).

4.3 Full GPT forward pass​

1. Token embedding lookup.
2. Add positional embedding.
3. Embedding dropout.
4. Stack of transformer blocks.
5. Final LayerNorm.
6. Linear head → logits [B, T, vocab_size].

Weight tying (optional): share token embedding weights with the output head to reduce parameter count (classic GPT-2). Separate matrices can train better; match the scheme when loading OpenAI weights.

4.4 Parameter and memory hygiene​

Rough parameter drivers:

• Embedding tables: vocab_size × d_model (vocab ~50k for GPT-2 BPE).
• Per block: QKV, out proj, FFN (d→4d→d), norms.
• Depth × width dominate at scale.

RAM estimate (float32): params × 4 bytes for weights alone; training needs optimizer states (AdamW ≈ 2× more) + activations.

4.5 Greedy text generation loop​

while tokens_to_generate:
  crop context to last context_length tokens
  logits = model(context)
  take last time step logits
  next_id = argmax(softmax(logits))   # or sample
  append next_id

Untrained weights → gibberish. That is expected and useful as a smoke test of shapes/devices.

Negative cases:

• Generating with dropout still enabled.
• Not cropping context → positional index OOB.
• Softmax on wrong dimension.
• Comparing train loss across different tokenizers/vocabs.

Part II — Stage 2: Pretraining the foundation model

5. Stage 2.1 — Loss and evaluation for generative models​

5.1 From logits to loss​

For each position, the model outputs a vocab_size-way distribution. Cross-entropy against the true next-token ID is the training objective.

Batching note: flatten [B, T, V] logits and [B, T] targets to [B·T, V] and [B·T] for cross_entropy.

Perplexity: exp(mean_loss). Intuition: effective uncertainty over vocabulary choices; lower is better. Do not over-interpret absolute numbers across different tokenizers.

5.2 Train / validation splits​

Even on tiny educational corpora:

1. Split text before windowing (avoid leakage across the cut).
2. Compute mean loss over loaders periodically.
3. Watch for train loss ↓ while val loss stagnates → overfitting / memorization (common on tiny books trained many epochs).

5.3 What “good” looks like during teaching runs​

On a short story, expect:

• Early epochs: loss falls quickly; samples become grammatical-ish.
• Later epochs: near-zero train loss with memorized passages; val loss worse → stop or use more data / regularization.

Production pretraining uses vastly more data, usually ~1 epoch over huge corpora, careful dedup, and serious eval suites (not only CE on one book).

6. Stage 2.2 — The training loop​

Canonical PyTorch loop for LM pretraining:

1. model.train()
2. For each batch: zero grad → forward → loss → backward → optimizer step
3. Periodically: evaluate() on train/val subsets
4. Periodically: generate a fixed prompt sample for qualitative inspection
5. Checkpoint model.state_dict() (+ optimizer if resuming)

Optimizer: AdamW is the common default for LLMs (decoupled weight decay).

Device: move model and batches to the same device (cuda / mps / cpu). Never mix devices in one op.

6.1 Optional training stabilizers (production-flavoured)​

Educational loops can omit these; longer runs benefit from them.

6.2 Checkpointing​

Save:

• Config (vocab, context, depth, heads, dims, dropouts, bias flags)
• Weights
• Optimizer state (if continuing training)
• Step / epoch / RNG seeds when reproducibility matters

Load into an identically configured module before load_state_dict.

7. Stage 2.3 — Decoding strategies (beyond greedy)​

Greedy argmax is deterministic and often repetitive.

Temperature: divide logits by T before softmax.

• T → 0 — peaky, near-greedy
• T = 1 — unmodified
• T > 1 — flatter, more random (can become nonsense)

Top-k: keep only the k largest logits; set others to -inf before softmax. Prevents sampling absurd low-probability tokens when temperature is high.

Combined recipe (common): top-k filter → temperature scale → multinomial sample. Optionally stop on EOS.

Application guidance:

• Low T / low k — factual QA, code, formal docs
• Higher T / higher k — brainstorming, fiction (accept quality variance)

Negative cases:

• Temperature ≤ 0
• Top-k larger than vocab
• Sampling while gradients are tracked (wasteful); use torch.no_grad()

8. Stage 2.4 — Loading open pretrained weights​

Pretraining GPT-3-class models is expensive. For learning and fine-tuning practice:

1. Implement architecture matching the checkpoint family (GPT-2 sizes: ~124M / 355M / 774M / 1.5B+).
2. Download official weight files (historically TensorFlow checkpoints; community loaders convert to PyTorch).
3. Map names carefully (c_attn splits into Q/K/V, LN scale/bias naming, etc.).
4. Enable whatever quirks the checkpoint expects (e.g. QKV bias on/off, context length 1024).
5. Smoke-test: coherent completion on a simple prompt.

Mismatch symptoms: immediate garbage text, shape errors in assign, or NaNs. Fix mapping before fine-tuning.

Legal / ops note: respect model licences and dataset rights; educational public-domain text ≠ licence to redistribute proprietary weights or copyrighted books.

Part III — Stage 3: Fine-tuning

9. Classification fine-tuning vs instruction fine-tuning​

Choose classification when the product decision is a closed label set. Choose instruction tuning when users issue varied natural-language tasks.

10. Stage 3.1 — Fine-tuning for classification (worked pattern)​

Example task: spam vs ham SMS.

10.1 Data preparation​

1. Load labeled rows.
2. Balance classes if severely skewed (undersample / other strategies).
3. Map labels to integers starting at 0.
4. Split train / val / test (e.g. 70/10/20) with a fixed seed.
5. Tokenize with the same tokenizer as the base model.
6. Pad to max length in training set (or model max); truncate longer texts.
7. Batch with DataLoader.

Padding token: GPT-2 style often uses end-of-text ID as pad; mask pads in attention if you implement explicit padding masks (implementation-dependent).

10.2 Model surgery​

1. Load pretrained GPT.
2. Replace out_head: d_model → vocab with d_model → num_classes.
3. Freeze most parameters; optionally unfreeze final block + final norm + new head (strong baseline on small data).
4. Train with CE on last token logits (causal attention makes the final position the richest summary for classification).

Why last token: under causal masking, only the last position can attend to the full prefix.

10.3 Metrics and loops​

• Track CE loss and accuracy on train/val.
• Early stop on val loss / accuracy.
• Report final test accuracy once.
• Persist state_dict for deployment experiments.

Negative cases:

• Fine-tuning first-token logits instead of last → large accuracy drop.
• Padding all sequences to 1024 “because we can” → slower, sometimes worse on short SMS.
• Evaluating with dropout on.
• Label leakage in preprocessing.
• Using a different tokenizer than the checkpoint.

10.4 Inference API shape​

Encode → pad/truncate → forward → argmax on last position → map to label string. Keep max_length consistent with training.

11. Stage 3.2 — Fine-tuning to follow instructions​

11.1 Dataset format​

Each example typically has:

• instruction
• optional input
• output (desired response)

Apply a prompt template (Alpaca-style and Phi-style are common variants). Consistency beats perfection—train and evaluate with the same template.

11.2 Batching for SFT (supervised fine-tuning)​

Custom collate usually:

1. Format full string = prompt + response.
2. Tokenize.
3. Pad to max length in the batch (or global max).
4. Targets = inputs shifted by one.
5. Set padding positions in targets to ignore_index=-100 so CE skips them.
6. Optionally mask instruction tokens in the target so loss focuses on the response only (research is mixed; measure on your eval).

11.3 Base model choice​

Very small bases struggle at instruction following. Educational practice often moves from GPT-2 small (~124M) to medium (~355M) when doing SFT on a laptop/GPU.

11.4 Training​

Reuse the LM training loop with instruction batches. Monitor:

• Train/val CE
• Qualitative generations on held-out instructions
• Overfitting on tiny instruction sets (memorized answers)

11.5 Evaluation is harder than spam accuracy​

Useful layers:

1. Spot checks — read model vs gold responses.
2. Automatic scoring — another LLM judges with a rubric (0–100), average over test set.
3. Structured benchmarks — MMLU-like suites for knowledge (different from conversational quality).
4. Human preference — slow, high signal.

Local judges (e.g. Ollama-hosted models) help classrooms without cloud APIs; scores may be nondeterministic—average multiple runs.

11.6 Parameter-efficient fine-tuning (LoRA)​

Instead of updating all weights, learn low-rank adapters ΔW ≈ BA on linear layers. Benefits:

• Fewer trainable parameters
• Smaller checkpoints per task
• Often faster iteration on large bases

Still validate quality; LoRA rank/alpha are hyperparameters.

12. End-to-end checklist (engineering)​

Stage 1 gates​

• Token round-trip: decode(encode(text)) sane for BPE
• Batch shapes [B, T] for inputs/targets; targets shifted
• Causal mask verified (no future attention)
• One transformer block preserves [B, T, d]
• Full model logits [B, T, V]
• Untrained generate runs without shape/device errors

Stage 2 gates​

• Train loss trends down
• Val loss computed on held-out text
• Samples improve then may memorize on tiny data
• Checkpoints reload and continue or generate
• Optional: open weights produce coherent English

Stage 3 gates​

• Classification: balanced data, last-token CE, test metrics
• Instruction: template consistency, pad masking, qualitative + scored eval
• Saved artifacts include config + tokenizer identity + template version

13. Failure modes worth designing for​

14. Production and enterprise caveats (playbook lens)​

Educational GPT builds are not production platforms. When you carry skills into client work:

• Data rights: pretraining/fine-tuning corpora need licence and privacy review.
• Eval before demo: offline metrics + red-team prompts + cost/latency budgets.
• Security: prompt injection, data exfiltration, insecure tool wiring—see OWASP LLM risks.
• Ops: version model + tokenizer + prompt template together; monitor drift and cost.
• Architecture choice: often use a managed frontier/regional model with RAG rather than pretrain from scratch—unless differentiation truly requires custom weights.
• Docker / networking: never publish database ports to the public internet; keep internal services on private networks.

Custom domain models can outperform general APIs on narrow tasks (finance, clinical QA patterns in literature)—but only with disciplined data, eval, and governance.

15. Suggested learning sequence (hours → days)​

1. PyTorch tensors, autograd, Dataset/DataLoader
2. BPE encode/decode + sliding-window batches
3. Implement attention → causal → multi-head
4. Assemble GPT; greedy generate (expect gibberish)
5. Pretrain on a tiny corpus; plot losses; sample
6. Load GPT-2 weights into your class
7. Spam classifier fine-tune
8. Instruction SFT + simple LLM-as-judge eval
9. Optional: LoRA, cosine schedule, larger data

Primary code companion: rasbt/LLMs-from-scratch. Primary narrative companion: Raschka, Build a Large Language Model (From Scratch), Manning (ISBN 9781633437166).

16. How this maps to the playbook​

Building from scratch trains intuition. Shipping enterprise value usually combines that intuition with hosted models, retrieval, evaluation harnesses, and governance—not a single from-scratch pretrain on a laptop.