Text generation

Use model.generate for LLMs and VLMs when you need decoding control, streaming or batch generation beyond Pipeline defaults.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/llm_tutorial. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: llm_tutorial, generation_features, perplexity

Executive summary

Tokenize → generate → decode. Set max_new_tokens, choose a decoding strategy, and enable streaming when UX needs tokens as they arrive.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "openai-community/gpt2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

inputs = tok("Transformers changed NLP because", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

Generation features

Hugging Face documents streaming, logits processors, stopping criteria, assisted decoding and more under Generation features. Pair with Decoding strategies and Optimisation.

Perplexity

For fixed-length LMs, perplexity remains a classic intrinsic metric — see Perplexity. For chat/instruction models, prefer task evals and LLM-as-judge suites.

Key takeaways

• generate is the core LLM API under Pipeline.
• Bound length; pick sampling vs greedy deliberately.

What to do next

Decoding strategies.