Inference optimisation

Squeeze latency and memory with attention backends, KV cache strategies, torch.compile, continuous batching and parallelism.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/optimization_overview. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: optimization_overview, attention_interface, kv_cache, cache_explanation, continuous_batching, paged_attention, perf_torch_compile, perf_infer_gpu_multi, llm_tutorial_optimization

Executive summary

Start with half/bfloat16 + FlashAttention/SDPA, tune KV cache, then consider compile, continuous batching or an external engine (vLLM) for high QPS.

High-impact levers

Lever	What it does
Dtype	float16 / bfloat16 cut memory and boost throughput
Attention backends	SDPA / FlashAttention for faster attention
KV cache	Avoid recomputing past keys/values; choose cache implementation
torch.compile	Fuse kernels for steady-state speedups
Continuous batching	Keep GPUs busy across concurrent requests
Paged attention	Memory-efficient KV management for serving
Tensor / expert parallelism	Shard large models across GPUs

Official entry: Optimisation overview.

KV cache

# generate uses KV cache by default for decoder-only models
out = model.generate(**inputs, max_new_tokens=128, use_cache=True)

Deep dive: Cache strategies and Caching.

When to leave Transformers eager serving

If you need OpenAI-compatible HTTP, high concurrency and paged attention at scale, deploy vLLM / SGLang while keeping the same Hub checkpoint.

Key takeaways

• Precision + attention backend + KV cache are the first wins.
• Continuous batching belongs in a server, not a notebook loop.

What to do next

Chat and tools or Serving.