Tokenizers

Tokenizers map text to integer ids (and back). Transformers prefers fast tokenizers from the 🤗 Tokenizers library when available.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/fast_tokenizers. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: fast_tokenizers, custom_tokenizers, tokenizer_summary

Executive summary

Use the tokenizer shipped with the checkpoint. Prefer fast tokenizers for speed and offset mappings. Match padding, truncation and special tokens to how the model was trained.

Load a tokenizer

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
encoded = tokenizer(
    "Hugging Face Transformers",
    padding=True,
    truncation=True,
    return_tensors="pt",
)
print(encoded.keys())  # input_ids, attention_mask, ...

Algorithms (summary)

Algorithm	Typical models	Idea
WordPiece	BERT	Merge frequent pieces; uses ## continuation
BPE	GPT-2, many LLMs	Byte-pair merges on frequency
Unigram	T5 / SentencePiece	Probabilistic subword set
Byte-level BPE	GPT-2 / RoBERTa lineage	Operates on bytes for full coverage

Official detail: Tokenization algorithms.

Fast vs slow

• Fast — Rust implementation; batching, offsets, alignment helpers
• Slow — pure Python fallback for some models

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)

Chat templates

Instruction models expect messages rendered through a chat template (Jinja). See Chat and tools.

Pitfalls

• Different tokenizers ⇒ different ids for the “same” text — never mix across models.
• Left vs right padding matters for decoder-only generation.
• add_eos_token / pad_token may need explicit setup for GPT-style models.

Key takeaways

• Always use the tokenizer paired with the weights.
• Prefer fast tokenizers; learn BPE vs WordPiece vs Unigram at a high level.
• Chat models need apply_chat_template.

What to do next

Vision and audio processors.