Quantization concepts and selection

Quantization stores weights (and sometimes activations) in lower precision to cut memory and often improve throughput — with quality trade-offs.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/quantization/overview. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: quantization/overview, quantization/selecting, quantization/concept_guide

Source: Hugging Face Transformers documentation.

Executive summary

Start from the Hugging Face selection guide: pick methods that match your hardware (CUDA, CPU, Metal) and whether you need training-aware, post-training, or runtime packing.

Mental model

Idea	Meaning
Weight-only	Weights low-bit; activations higher precision
PTQ	Quantize after training
QAT	Train aware of quantization
Calibration	Representative data to set scales
Packing	Dense storage of low-bit weights

Source: Hugging Face Transformers documentation.

Source: Hugging Face Transformers documentation.

Official: Overview, Selecting a method, Concept guide.

Source: Hugging Face Transformers documentation.

Key takeaways

• Quantization is a measured trade-off — always eval after compressing.
• Hardware support dictates feasible methods.

What to do next

Core methods.