Processors and multimodal inputs

A Processor bundles the text tokenizer with image/audio/video preprocessors so multimodal models receive a single aligned batch.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/processors. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: processors, multimodal_processing, feature_extractors

Executive summary

For VLMs and speech-text models, load AutoProcessor (not only the tokenizer). It keeps text and media preprocessing versioned with the weights.

from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image

processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = AutoModelForVision2Seq.from_pretrained("Salesforce/blip-image-captioning-base")

image = Image.open("cat.jpg")
inputs = processor(images=image, text="a photo of", return_tensors="pt")
outputs = model.generate(**inputs)
print(processor.decode(outputs[0], skip_special_tokens=True))

When to use what

Class	Modality
AutoTokenizer	Text
AutoImageProcessor	Images
AutoFeatureExtractor	Audio (and some vision legacy APIs)
AutoProcessor	Multimodal combinations
AutoVideoProcessor	Video

Multimodal chat

Many VLMs use chat templates that embed image/audio placeholders. See Chat and tools and Hugging Face’s multimodal chat templates.

Key takeaways

• Multimodal = processor, not tokenizer-alone.
• Keep processor and model revisions in lockstep.

What to do next

Pipeline for task-level multimodal inference.