Vision Transformer (ViT)

ViT adapts Transformers to vision by splitting an image into fixed-size patches treated as tokens.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/model_doc/vit. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: model_doc/vit

Source: Hugging Face documentation images.

Executive summary

ViT is the baseline vision backbone in Transformers. Fine-tune for classification or use as a backbone in detection/segmentation stacks.

Quick usage

from transformers import pipeline

clf = pipeline(task="image-classification", model="google/vit-base-patch16-224")
print(clf("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"))

from transformers import AutoImageProcessor, AutoModelForImageClassification
from PIL import Image
import requests

processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224", use_fast=True)
model = AutoModelForImageClassification.from_pretrained("google/vit-base-patch16-224")
image = Image.open(requests.get(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg",
    stream=True,
).raw)
inputs = processor(images=image, return_tensors="pt")
logits = model(**inputs).logits

Official API

ViT model doc.

What to do next

CLIP.