CLIP

CLIP jointly trains image and text encoders on large image–text pairs so embeddings share a space — enabling zero-shot image classification via text prompts.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/model_doc/clip. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: model_doc/clip

Source: Hugging Face documentation images.

Executive summary

Use CLIP for zero-shot labels, retrieval (image↔text) and as a building block in multimodal systems. Prompt phrasing (“a photo of …”) strongly affects results.

Quick usage

from transformers import pipeline

clip = pipeline(task="zero-shot-image-classification", model="openai/clip-vit-base-patch32")
labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
print(clip("http://images.cocodataset.org/val2017/000000039769.jpg", candidate_labels=labels))

Official API

CLIP model doc.

What to do next

Qwen2.