Parameter-efficient fine-tuning (PEFT)

PEFT methods (especially LoRA) adapt large models by training small adapter weights instead of all parameters.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/peft. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: peft

Executive summary

Use PEFT/LoRA when full fine-tuning exceeds budget or when you need many task adapters on one base model. Merge adapters for deployment when appropriate.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
config = LoraConfig(r=16, lora_alpha=32, target_modules=["c_attn"], lora_dropout=0.05, bias="none")
model = get_peft_model(model, config)
model.print_trainable_parameters()

Follow the official PEFT guide and the standalone PEFT docs. Ecosystem trainers: TRL / Unsloth / Axolotl.

Enterprise benefits

• Lower GPU memory and faster iteration
• Adapter-per-tenant or adapter-per-task packaging
• Easier rollback (swap adapters without replacing the base)

Key takeaways

• LoRA is the default PEFT choice for LLMs.
• Track base revision + adapter revision as a pair.

What to do next

Memory and speed.