BERT

BERT is a bidirectional Transformer encoder pretrained with masked language modelling and next-sentence prediction. Learned representations transfer well via fine-tuning heads.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/model_doc/bert. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: model_doc/bert

Executive summary

Use BERT-family encoders for classification, NER, extractive QA and embedding-style features. Prefer modern successors (RoBERTa, ModernBERT, etc.) when quality matters, but BERT remains the teaching baseline.

Why it matters

• Bidirectional context (vs GPT-style causal masking)
• Versatile: add a head and fine-tune
• Hub collection: google/bert-release

Quick usage

from transformers import pipeline

pipe = pipeline(task="fill-mask", model="google-bert/bert-base-uncased")
print(pipe("Plants create [MASK] through a process known as photosynthesis."))

from transformers import AutoModelForMaskedLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained(
    "google-bert/bert-base-uncased",
    device_map="auto",
    attn_implementation="sdpa",
)

Fine-tuning tips

• Use AutoModelForSequenceClassification / TokenClassification / QuestionAnswering
• Uncased vs cased tokenizers matter for NER
• See NLP tasks

Official API

Full class reference: BERT model doc.

What to do next

GPT-2 for decoder-only generation.