Distributed training

When one GPU is not enough, scale with data parallel, sharded optimisers or model parallel techniques.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/accelerate. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: accelerate, accelerator_selection, ddp, fsdp, deepspeed, deepspeed_alst, tensor_parallelism, debugging, perf_train_gpu_many

Options (overview)

Method	Summary
Accelerate	Unified launcher / device helpers
DDP	Classic data parallel
FSDP2	Shard parameters/gradients/optimiser states
DeepSpeed ZeRO	Memory-efficient sharded training
Tensor parallelism	Split layers across GPUs
Ulysses SP	Sequence parallelism for long context

Start from Accelerate and Parallelism methods.

Debugging

Distributed failures are often NCCL, topology or batch-size issues. Use Hugging Face’s Debugging notes and keep a single-GPU baseline.

Key takeaways

• Prove the recipe on 1 GPU, then scale.
• Match parallelism strategy to model size and sequence length.

What to do next

Hardware.