Customising models

Adapt pretrained architectures by changing configuration, swapping heads, composing modular blocks, or carefully patching forward passes.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/custom_models. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: custom_models, monkey_patching, fusion_mapping, how_to_hack_models, modular_transformers

Executive summary

Start with config edits and classification/LM heads. Use modular Transformers when contributing reusable blocks. Reserve monkey-patches for short experiments — promote lasting changes into proper modelling code.

Common customisation paths

1. Config overrides — change num_labels, dropout, max positions before from_pretrained.
2. Task heads — load a backbone then attach AutoModelFor* heads for classification, QA, etc.
3. Modular shards — declare reuse in modular_*.py (contributor workflow).
4. Component hacks — attention backends, fusion maps, intermediate tracing (advanced).

from transformers import AutoConfig, AutoModelForSequenceClassification

config = AutoConfig.from_pretrained("bert-base-uncased", num_labels=4)
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    config=config,
)

Guidance

Approach	When
Config + head	Fine-tuning for a new label set
PEFT (LoRA)	Efficient adaptation without full weight updates — see PEFT
Modular contribution	Upstreaming a new architecture
Monkey patch	Spike / debug only

Official deep dives: Custom models, Monkey patching, Modular transformers.

Key takeaways

• Prefer config and head changes over forking modelling files.
• PEFT covers most enterprise adaptation needs.
• Upstream modular shards when sharing architectures.

What to do next

Sharing and export.