Inference engines

Production LLM serving usually moves from eager PyTorch to specialised engines that still consume Hugging Face model definitions.

Adapted for this playbook from the 🤗 Transformers documentation by Hugging Face. Official page: https://huggingface.co/docs/transformers/community_integrations/vllm. Images © Hugging Face (documentation-images) unless noted. This is not a substitute for the upstream docs — verify against the current version.

Covers Hugging Face pages: community_integrations/vllm, sglang, tensorrt-llm, transformers_as_backend

Executive summary

Keep checkpoints in Hub/Transformers format; serve with vLLM or SGLang for continuous batching and high throughput. Use TensorRT-LLM when the NVIDIA stack is standardised.

Engine	Official Transformers integration doc
vLLM	vLLM
SGLang	SGLang
TensorRT-LLM	TensorRT-LLM
Backend guide	Building a compatible backend

Why this split exists

• Transformers optimises for correct, complete model definitions and researcher ergonomics
• Engines optimise for scheduler, KV memory, batching and kernels

Enterprise pattern: train/adapt with Transformers + PEFT → export or load the same repo in vLLM → put API gateway, auth and eval harness in front.

Key takeaways

• Do not rewrite models to use an engine — use HF-compatible loaders.
• Benchmark TTFT and tokens/s on your traffic shape.

What to do next

Local deployment.