Decoding strategies

Decoding converts next-token distributions into token sequences. Strategy choice changes creativity, factuality and latency.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/generation_strategies. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: generation_strategies, tasks/prompting, assisted_decoding

Common strategies

Strategy	Flags (typical)	Use when
Greedy	do_sample=False	Deterministic, short answers
Beam search	num_beams>1	Higher likelihood sequences (translation/summarisation)
Temperature sampling	do_sample=True, temperature	Creative text
Top-k / top-p	top_k, top_p	Truncate tail of distribution
Assisted decoding	assistant model	Speculative speed-ups

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

Prompting

Decoding cannot fix a bad prompt. See Hugging Face’s Prompt engineering task guide and this playbook’s Prompt & context engineering.

Assisted decoding

Speculative / assisted decoding uses a smaller draft model to propose tokens verified by the target model — see Assisted decoding.

Key takeaways

• Match strategy to task (beam for seq2seq; sampling for open chat).
• Constrain length and stop tokens in production.

What to do next

Optimisation.