Memory and speed

Training bottlenecks are usually memory first, then throughput. Hugging Face documents both levers extensively.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/model_memory_anatomy. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: model_memory_anatomy, grad_accumulation, grad_checkpointing, mixed_precision_training, torch_compile, kernels, padding_free

Memory

Technique	Idea
Gradient accumulation	Simulate larger batches
Gradient checkpointing	Recompute activations to save VRAM
Mixed precision	FP16 / BF16
PEFT / quantization	Fewer trainable or lower-bit weights
Optimiser choice	8-bit Adam, etc.

Speed

Technique	Idea
torch.compile	Kernel fusion
Fused kernels	Attention / MLP kernels
Padding-free training	Pack sequences to cut waste
DataLoader workers / pinning	Keep GPU fed

Official: GPU memory usage, Mixed precision, torch.compile.

Key takeaways

• Measure allocation before changing three knobs at once.
• BF16 on modern NVIDIA GPUs is often the best default.

What to do next

Distributed training.