Knowledge Distillation for Large Language Models

Knowledge distillation is a model-compression and capability-transfer technique in which a powerful teacher model provides training signals for a smaller student model.

Instead of requiring the student to rediscover every useful behaviour from raw internet-scale pretraining data, the student learns from the teacher’s outputs, probability distributions, internal representations, reasoning demonstrations or preferences.

A distilled model can therefore become:

• Faster at inference
• Less expensive to operate
• Smaller in memory
• Easier to deploy on limited hardware
• More specialised for a particular task
• More consistent than the original general-purpose model for a narrow workflow

However, an important distinction must be made:

The exact internal process used to train current proprietary Claude models is not publicly disclosed in full.

Anthropic publishes model reports, safety research and selected training information, but its public materials do not expose Claude’s weights, token logits, hidden states, training datasets or the complete teacher–student training recipe. Therefore, when people discuss “distilling Claude,” they may be referring to two different things:

1. Authorised distillation within an official platform, such as Amazon Bedrock’s documented Claude Sonnet-to-Haiku distillation workflow.
2. Black-box behavioural distillation, where permitted Claude outputs are collected and used to fine-tune another model.

Anthropic and AWS have publicly described an authorised workflow in which Claude 3.5 Sonnet generates synthetic training data, Claude 3 Haiku is trained and evaluated using that data, and the resulting distilled model is hosted for inference. This is primarily output-based behavioural distillation rather than traditional access to Claude’s internal logits or hidden layers.

1. The fundamental idea​

Consider a large teacher model:

where:

• (x) is the prompt or input
• (y) is the generated output
• (p_T) is the teacher’s probability distribution
• (T) identifies the teacher model

The smaller student model has its own distribution:

where (\theta) represents the trainable parameters of the student.

The objective is to find parameters (\theta) such that:

for the inputs and tasks that matter.

The goal is not necessarily to copy every fact or behaviour of the teacher. Usually, the practical objective is narrower:

while:

and:

The classical knowledge-distillation framework was formalised by Hinton, Vinyals and Dean. Their key insight was that a teacher’s complete probability distribution contains more information than a single correct label.

2. What “knowledge” is being distilled?​

Knowledge in a neural model is not stored as a normal database of sentences. It is distributed across:

• Model parameters
• Attention patterns
• Intermediate activations
• Token probability relationships
• Learned concepts
• Decision boundaries
• Response strategies
• Tool-selection behaviours
• Safety and refusal behaviours
• Formatting conventions
• Reasoning patterns
• Domain-specific language

Knowledge distillation can transfer different subsets of this knowledge.

2.1 Final-answer knowledge​

The teacher provides a final answer:

Prompt:
Classify this customer message as billing, technical support or cancellation.

Teacher response:
billing

The student learns:

This is the simplest form of behavioural distillation.

2.2 Soft probability knowledge​

Suppose the possible classes are:

• Billing
• Technical support
• Cancellation
• Sales

The human label might only say:

Billing = 1
Everything else = 0

But the teacher might produce:

This tells the student that:

• Billing is most likely.
• Technical support is somewhat plausible.
• Cancellation is less plausible.
• Sales is very unlikely.

This relationship among wrong and correct alternatives is sometimes called dark knowledge. It helps the student understand how concepts relate rather than merely memorising the correct class.

2.3 Sequence-generation knowledge​

For an LLM, the teacher provides an entire sequence:

The sequence probability is:

The student learns not only which words are appropriate but also:

• How to structure an answer
• Which information to mention first
• How long the response should be
• How explanations are organised
• How instructions are followed
• When tools should be called
• How JSON or code should be formatted

Sequence-level distillation was explored in machine translation by training a student on complete sequences selected or generated by a stronger teacher.

2.4 Reasoning-pattern knowledge​

The teacher can provide an explanation or a verifiable intermediate derivation:

Question:
A service receives 4,000 requests per minute. A new caching layer removes
65% of repeated requests. How many requests reach the model?

Rationale:
Remaining proportion = 1 - 0.65 = 0.35.
Requests reaching the model = 4,000 × 0.35 = 1,400.

Answer:
1,400 requests per minute.

The student can be trained on:

where:

• (r) is the rationale
• (y) is the final answer

Research such as Distilling Step-by-Step showed that generated rationales can provide richer supervision than labels alone. Its multi-task formulation trained the student separately to predict the rationale and the label.

This should not be confused with extracting a provider’s private internal chain of thought. In a legitimate system, training should use outputs that the provider intentionally returns and that the customer is authorised to use.

2.5 Internal representation knowledge​

When the teacher weights are available, the student may imitate:

• Hidden states
• Attention matrices
• Query–key relationships
• Value relationships
• Embeddings
• Intermediate feature maps

For teacher layer (l):

and student layer (m):

a representation loss may be defined as:

The projection matrix (W) is used when the teacher and student have different hidden dimensions.

DistilBERT combined language-model loss, distillation loss and a cosine-distance loss between representations. The reported model reduced BERT’s size by 40%, retained 97% of its language-understanding capability on the study’s evaluations and was 60% faster.

This type of internal distillation is generally not possible with black-box APIs such as standard Claude or GPT API access.

3. The three principal access regimes​

3.1 Black-box distillation​

The distiller can observe only the teacher’s inputs and outputs:

Available signals may include:

• Generated text
• Structured JSON
• Tool calls
• Code
• Classifications
• Explanations intentionally returned by the model
• Scores requested from the model
• Multiple sampled answers

Unavailable signals usually include:

• Full vocabulary logits
• Hidden-state tensors
• Attention matrices
• Gradients
• Model weights
• Private internal reasoning
• Optimiser state

Distilling from a standard Claude API response is therefore usually black-box behavioural distillation.

The common training objective is supervised fine-tuning:

The student is rewarded for reproducing the teacher-generated sequence.

This is sometimes called:

• Response distillation
• Synthetic-data fine-tuning
• Imitation learning
• Sequence-level distillation
• API-based distillation

These terms overlap but are not always identical.

3.2 Grey-box distillation​

The distiller receives more information than text but less than full model access.

Possible signals include:

• Top-(k) token probabilities
• Log-probabilities for generated tokens
• Confidence scores
• Teacher reward scores
• Multiple ranked responses
• Embeddings
• Task-specific representations

For example, the provider might return the top five next-token probabilities rather than the entire vocabulary distribution.

This enables an approximate logit-distillation objective while reducing data size.

3.3 White-box distillation​

The teacher’s weights and internal tensors are available.

The training process can use:

• Full token logits
• Hidden-state matching
• Attention matching
• Embedding matching
• Layer-to-layer mapping
• Gradient-based teacher feedback
• On-policy interaction between teacher and student

Open-weight model families such as compatible Llama, Qwen, Gemma or other licensed models are more suitable for this approach, subject to their licences.

Google’s Gemma terms explicitly describe model derivatives as including models created through transfer of weights, internal representations or synthetic outputs using distillation. This illustrates why model licences must be checked rather than assuming that “open weights” means unrestricted use.

4. How Claude distillation is done in practice​

4.1 What is publicly documented​

Anthropic and AWS described a managed Claude distillation process with the following high-level workflow:

1. A large Claude teacher, such as Claude 3.5 Sonnet, generates high-quality responses.
2. Data-synthesis techniques expand and improve the training dataset.
3. A smaller Claude student, such as Claude 3 Haiku, is fine-tuned.
4. The student is evaluated.
5. The resulting private model is hosted for inference.

AWS states that Bedrock can create teacher responses from submitted prompts or reuse responses stored in model-invocation logs. It then augments the data, separates training and validation sets, fine-tunes the student and makes the resulting custom model available only to the customer.

Conceptually:

This is a strong enterprise pattern because the customer does not need direct access to Claude’s weights or logits.

4.2 What is not publicly documented​

The public description does not reveal every underlying training detail. For example, it does not fully disclose:

• Whether all jobs use token-level teacher distributions
• The exact loss-function mixture
• Internal layer-matching objectives
• Optimiser configurations
• The precise data-synthesis algorithms
• Teacher sampling temperatures
• Teacher ensembles
• Detailed filtering thresholds
• All safety-transfer mechanisms
• Complete evaluation datasets

Therefore, one should not claim that every Claude Haiku model is simply a compressed Sonnet model or that the entire Claude family is produced using one known distillation method.

Anthropic’s public model reports describe broad pretraining and post-training processes, but the complete proprietary architecture and training pipeline are not released.

4.3 Why a Claude teacher can improve a smaller student​

Assume Claude Sonnet can already perform a customer-service workflow well:

A smaller student may initially fail because it:

• Misclassifies ambiguous requests
• Omits policy conditions
• Produces invalid JSON
• Escalates too often
• Escalates too rarely
• Uses an inconsistent tone
• Hallucinates unavailable products

The teacher can generate thousands of examples covering:

• Common cases
• Edge cases
• Adversarial wording
• Multilingual variants
• Incomplete questions
• Conflicting instructions
• Tool-use decisions
• Refusal conditions
• Escalation criteria

The student then learns the teacher’s operational pattern.

This does not make the student globally equivalent to Claude Sonnet. It can make the student highly competitive within the selected distribution.

5. Classical logit-based knowledge distillation​

5.1 Teacher and student logits​

For an input (x), suppose the teacher produces logits:

and the student produces:

where (V) is the vocabulary size.

The ordinary softmax is:

Knowledge distillation introduces a temperature (\tau):

When (\tau>1), the probability distribution becomes softer.

For example:

Temperature 1​

Higher temperature​

The softer distribution exposes more information about the teacher’s alternatives.

5.2 KL-divergence loss​

A common teacher–student loss is:

where:

The (\tau^2) term compensates for gradient scaling caused by the temperature.

The student may also learn from the original ground-truth labels:

A combined objective is:

where (0\leq\alpha\leq1).

The teacher provides nuanced behavioural information, while the original labels prevent the student from blindly inheriting every teacher error.

5.3 Token-level LLM distillation​

For causal language models, the distillation objective is applied at every output position:

The student learns the teacher’s distribution over the entire vocabulary for each token.

This is substantially richer than training only on the teacher’s selected token.

Suppose the teacher selected the token “London,” but its distribution was:

London      0.61
Manchester  0.14
Birmingham  0.08
England     0.06
Paris       0.01
Other       0.10

A normal synthetic-output dataset stores only “London.”

Logit distillation preserves the complete relationship.

6. Forward KL versus reverse KL​

6.1 Forward KL​

Classical distillation often minimises:

This is called forward KL in the teacher-to-student direction.

It strongly penalises the student for failing to cover modes that the teacher considers possible.

This can make the student mode-covering.

For language generation, however, the teacher may assign probability across many valid responses. A much smaller student may not have enough capacity to reproduce every mode well.

6.2 Reverse KL​

An alternative is:

This encourages the student to concentrate on high-probability regions of the teacher distribution.

It is generally more mode-seeking.

MiniLLM proposed reverse-KL distillation for generative language models and used on-policy sampling, teacher-mixed sampling, length normalisation and variance-reduction techniques to stabilise training. The authors reported improvements in response precision, calibration, exposure bias and long-generation quality compared with conventional baselines.

The practical intuition is:

• Forward KL says: “Cover all responses that the teacher might produce.”
• Reverse KL says: “Produce responses that the teacher considers highly likely.”

Neither is automatically best. The appropriate objective depends on:

• Student capacity
• Desired diversity
• Task type
• Generation length
• Safety constraints
• Whether mode collapse is acceptable

7. Sequence-level distillation​

When only teacher outputs are available, sequence-level distillation is the most common technique.

Given a prompt (x_i), the teacher generates:

or samples one or more high-quality outputs.

The distilled dataset is:

The student is trained using normal autoregressive cross-entropy:

This technique is easy to implement because the student does not need access to teacher internals.

However, one teacher answer per prompt may be insufficient. Better systems generate multiple candidates:

They then:

1. Verify correctness.
2. Remove low-quality outputs.
3. Rank the remaining answers.
4. Retain the strongest answer or several diverse answers.
5. Train the student on the curated set.

8. Rationale and reasoning distillation​

8.1 Direct answer distillation​

Loss:

This is efficient but may teach only surface behaviour.

8.2 Rationale-augmented distillation​

Loss:

The student receives information about how intermediate concepts connect to the final answer.

A multi-task structure can use prefixes:

The two objectives are:

The Distilling Step-by-Step study found that treating rationale prediction and label prediction as separate tasks performed better than simply concatenating rationale and answer into one target for the evaluated tasks.

8.3 Progressive explanation distillation​

The Orca project used large-scale explanation traces generated by GPT-4 and ChatGPT. It varied system instructions and progressively exposed a smaller model to complex explanations rather than using only short imitation responses.

The broader lesson is that teacher-data quality depends on more than the number of responses.

Useful diversity includes:

• Explanation depth
• Problem difficulty
• Response length
• Role instructions
• Domain
• Language
• Output format
• Tool availability
• Error-correction behaviour

A student trained only on polished final answers may imitate style without learning robust decision patterns.

8.4 Reasoning distillation in DeepSeek-R1​

DeepSeek reported generating a curated dataset of approximately 800,000 samples using DeepSeek-R1 and then directly supervised-fine-tuning smaller Qwen and Llama-family models on that dataset. The reported distilled models ranged from 1.5B to 70B parameters, and the distillation stage used SFT without an additional reinforcement-learning stage.

This is an important example because it demonstrates that “reasoning distillation” may be operationally simple:

It does not necessarily require hidden-state matching or sophisticated KL objectives.

9. A complete production distillation pipeline​

Phase 1: Define the target capability​

Do not begin with:

“We want a smaller Claude.”

Begin with measurable behaviour:

“We want a model that resolves Tier-1 billing questions with at least 92% policy-compliant accuracy, under a 1.5-second p95 latency target and a maximum operating cost of $0.004 per conversation.”

Define:

• Target users
• Task boundaries
• Required languages
• Input lengths
• Output formats
• Latency requirements
• Throughput requirements
• Cost ceiling
• Safety requirements
• Human-escalation rules
• Accuracy metrics
• Out-of-scope behaviour

A narrow objective is much easier to distil successfully than general intelligence.

Phase 2: Establish a baseline​

Evaluate:

1. The teacher model.
2. The unmodified student model.
3. A prompt-engineered student.
4. A student with RAG.
5. A student with RAG and tools.

Distillation may be unnecessary if the base student already meets the requirement.

AWS similarly recommends evaluating whether the selected student already performs adequately before creating a distilled model.

Phase 3: Construct the prompt distribution​

The distillation dataset must represent the real deployment distribution.

A customer-service dataset might include:

Sources may include:

• Approved production logs
• Human-authored prompts
• Synthetic edge cases
• Red-team cases
• Existing evaluation sets
• Policy documents converted into scenarios
• Domain-expert examples

Sensitive data should be removed, masked or appropriately governed before it is sent to a teacher or used for training.

Phase 4: Generate teacher responses​

For each prompt, request a structured response.

Example:

{
  "intent": "duplicate_charge",
  "requires_tool": true,
  "tool_name": "get_recent_transactions",
  "requires_escalation": false,
  "answer": "I can help check whether the charge was duplicated.",
  "policy_references": ["BILLING-04"],
  "confidence": 0.93
}

Structured outputs are easier to:

• Validate
• Filter
• Score
• Compare
• Train on
• Audit

Teacher generation should vary important dimensions:

• Temperature
• Prompt wording
• System instruction
• Input difficulty
• Context length
• Language
• Tool result
• Error condition

Avoid adding random diversity that would never occur in production.

Phase 5: Generate multiple candidates​

For difficult prompts, create (k) candidates:

Then score each candidate:

where:

• (C): correctness
• (F): format compliance
• (P): policy compliance
• (H): helpfulness
• (R): risk or hallucination penalty

Possible graders include:

• Deterministic rules
• Unit tests
• Schema validators
• Retrieval-grounding checks
• Domain experts
• A separate judge model
• Human review
• Tool-execution results

The strongest answer becomes the supervised target.

Phase 6: Filter and clean the teacher dataset​

Teacher outputs should never be accepted automatically at scale.

Filter for:

Correctness​

• Does the answer match the available evidence?
• Is the calculation correct?
• Does the code pass tests?
• Is the classification correct?

Grounding​

• Are factual claims supported by the source?
• Are policy references valid?
• Did the answer invent unavailable information?

Format​

• Is the JSON valid?
• Are required fields present?
• Does the tool call match the schema?
• Does the response obey the length limit?

Safety​

• Does the answer violate policy?
• Does it leak sensitive information?
• Does it follow malicious instructions in retrieved documents?

Diversity​

Remove:

• Exact duplicates
• Near duplicates
• Excessive template repetition
• Overrepresented easy cases
• Repeated teacher phrases

Contamination​

Ensure that evaluation examples have not been included in training.

Phase 7: Train the student​

The simplest black-box objective is:

Training may use:

• Full fine-tuning
• LoRA
• QLoRA
• Adapter tuning
• Prefix tuning
• Parameter-efficient fine-tuning

For highly specialised tasks, LoRA or QLoRA may be sufficient.

For broad behavioural transfer, full fine-tuning or continued pretraining may be more effective.

Phase 8: Add preference distillation​

The teacher can produce or rank multiple answers:

The student can then be trained using preference optimisation such as DPO.

A simplified DPO objective is:

This helps the student learn distinctions such as:

• Concise versus unnecessarily verbose
• Correct escalation versus over-escalation
• Grounded answer versus hallucinated answer
• Safe answer versus unsafe answer
• Valid tool call versus malformed tool call

Phase 9: Preserve general capabilities​

A narrow synthetic dataset may cause catastrophic forgetting.

The student might become excellent at billing classification but worse at:

• General language understanding
• Normal conversation
• Multilingual responses
• Safety
• Formatting
• Common-sense reasoning

Mix the distillation dataset with a replay dataset:

An equivalent loss is:

The correct ratio must be established experimentally.

Phase 10: Evaluate the distilled student​

Evaluation must compare at least:

• Teacher
• Base student
• Fine-tuned student
• Distilled student
• Production baseline

Capability metrics​

• Accuracy
• Exact match
• F1
• Pass@1
• Tool-call success
• JSON validity
• Retrieval faithfulness
• Task-completion rate

Behavioural metrics​

• Instruction adherence
• Policy compliance
• Escalation precision
• Escalation recall
• Tone
• Conciseness
• Refusal correctness

Operational metrics​

• p50 latency
• p95 latency
• Time to first token
• Tokens per second
• Memory usage
• GPU utilisation
• Cost per task
• Throughput

Safety metrics​

• Jailbreak resistance
• Prompt-injection resistance
• PII leakage
• Toxicity
• Harmful-completion rate
• Over-refusal
• Under-refusal

Robustness metrics​

• Misspellings
• Paraphrases
• Long context
• Conflicting evidence
• Missing data
• Tool failure
• Distribution shift

10. Implementing white-box logit distillation in PyTorch​

The following simplified function combines token-level KL divergence with ordinary language-model cross-entropy.

from __future__ import annotations

import torch
import torch.nn.functional as F
from torch import Tensor


def causal_lm_distillation_loss(
    student_logits: Tensor,
    teacher_logits: Tensor,
    labels: Tensor,
    *,
    temperature: float = 2.0,
    alpha: float = 0.8,
    ignore_index: int = -100,
) -> Tensor:
    """
    Compute a combined causal-language-model and knowledge-distillation loss.

    Shapes:
        student_logits: [batch, sequence_length, vocabulary_size]
        teacher_logits: [batch, sequence_length, vocabulary_size]
        labels:         [batch, sequence_length]

    alpha controls the balance:
        alpha = 1.0 -> teacher-distribution loss only
        alpha = 0.0 -> ground-truth cross-entropy only
    """
    if student_logits.shape != teacher_logits.shape:
        raise ValueError(
            "Teacher and student logits must have identical shapes. "
            "A vocabulary-mapping strategy is required when tokenizers differ."
        )

    if not 0.0 <= alpha <= 1.0:
        raise ValueError("alpha must be between 0 and 1.")

    if temperature <= 0:
        raise ValueError("temperature must be greater than zero.")

    # Shift because token t predicts token t+1.
    student_shifted = student_logits[:, :-1, :]
    teacher_shifted = teacher_logits[:, :-1, :].detach()
    labels_shifted = labels[:, 1:]

    valid_mask = labels_shifted.ne(ignore_index)

    student_log_probs = F.log_softmax(
        student_shifted / temperature,
        dim=-1,
    )

    teacher_probs = F.softmax(
        teacher_shifted / temperature,
        dim=-1,
    )

    teacher_log_probs = F.log_softmax(
        teacher_shifted / temperature,
        dim=-1,
    )

    # KL(P_teacher || P_student) for every token position.
    token_kl = torch.sum(
        teacher_probs * (teacher_log_probs - student_log_probs),
        dim=-1,
    )

    if valid_mask.any():
        kd_loss = token_kl[valid_mask].mean()
    else:
        kd_loss = token_kl.mean() * 0.0

    kd_loss = kd_loss * (temperature ** 2)

    vocabulary_size = student_shifted.size(-1)

    ce_loss = F.cross_entropy(
        student_shifted.reshape(-1, vocabulary_size),
        labels_shifted.reshape(-1),
        ignore_index=ignore_index,
    )

    return alpha * kd_loss + (1.0 - alpha) * ce_loss

A training step would conceptually look like:

teacher.eval()
student.train()

with torch.no_grad():
    teacher_outputs = teacher(
        input_ids=input_ids,
        attention_mask=attention_mask,
    )

student_outputs = student(
    input_ids=input_ids,
    attention_mask=attention_mask,
)

loss = causal_lm_distillation_loss(
    student_logits=student_outputs.logits,
    teacher_logits=teacher_outputs.logits,
    labels=labels,
    temperature=2.0,
    alpha=0.8,
)

optimizer.zero_grad(set_to_none=True)
loss.backward()
torch.nn.utils.clip_grad_norm_(student.parameters(), max_norm=1.0)
optimizer.step()

This requires teacher and student outputs to share a compatible vocabulary. If the tokenizers differ, the raw logits cannot be compared directly without a mapping strategy.

11. Distillation when teacher and student tokenizers differ​

Suppose the teacher uses vocabulary (V_T) and the student uses (V_S):

Then:

and:

A direct KL divergence is undefined because the dimensions and token meanings differ.

Possible solutions include:

11.1 Sequence-level distillation​

Generate teacher text and tokenise it with the student tokenizer.

This is the simplest approach.

11.2 Shared-vocabulary projection​

Map teacher token probabilities into a common vocabulary.

This is difficult because tokens may represent different byte sequences or subword units.

11.3 Word- or character-level matching​

Convert probabilities or outputs into a lower-level common representation.

This is computationally expensive and approximate.

11.4 Hidden-state alignment​

Project teacher and student representations into a common latent space:

This requires white-box access to both models.

11.5 Distribution matching over complete sequences​

Compare sequence probabilities or use a reward based on teacher likelihood rather than matching token vocabularies directly.

MiniLLM-style on-policy approaches are examples of this broader family.

12. Dataset format for black-box distillation​

A robust record should contain more than a prompt and answer.

{
  "example_id": "billing-004219",
  "task": "customer_service",
  "subtask": "duplicate_charge",
  "language": "en-GB",
  "difficulty": "medium",
  "prompt": "I have two identical charges from yesterday. What happened?",
  "teacher_response": {
    "intent": "duplicate_charge",
    "requires_tool": true,
    "tool_name": "get_recent_transactions",
    "requires_escalation": false,
    "answer": "I can help check whether the payment was processed twice."
  },
  "teacher_model": "approved-teacher-version",
  "generation_config": {
    "temperature": 0.2,
    "max_output_tokens": 500
  },
  "validation": {
    "schema_valid": true,
    "policy_valid": true,
    "human_reviewed": false,
    "grounding_score": 0.97
  },
  "data_governance": {
    "contains_personal_data": false,
    "consent_basis": "synthetic",
    "retention_class": "training-approved"
  }
}

Metadata supports:

• Data lineage
• Reproducibility
• Auditability
• Error analysis
• Data deletion
• Model-version comparison
• Regulatory reporting

13. Distilling tool use and agents​

Agentic distillation is more complicated because the desired target is not simply an answer. It is a trajectory.

A trajectory may contain:

where:

• (s_t): current state
• (a_t): action or tool call
• (o_t): tool observation
• (y): final answer

Example:

The student can be trained on:

• Tool-selection decisions
• Tool arguments
• Ordering of actions
• Error recovery
• Stop conditions
• Escalation decisions
• Final response synthesis

Agent distillation losses​

A combined objective might be:

where:

• \mathcal{L}_{action} teaches tool selection
• \mathcal{L}_{arguments} teaches valid tool parameters
• \mathcal{L}_{final} teaches answer generation
• \mathcal{L}_{safety} teaches policy-compliant behaviour

A student trained only on successful trajectories may not learn recovery. Include:

• Tool timeouts
• Empty search results
• Invalid responses
• Permission errors
• Conflicting sources
• Duplicate actions
• Partial failures

14. Distillation for RAG systems​

In a RAG system, the teacher should be given the same retrieved context that the student will receive.

The training example becomes:

where:

• (x): user question
• (c): retrieved context
• (y_T): grounded teacher answer

The student learns:

The dataset should contain:

• Correct retrieval results
• Partially relevant context
• Conflicting passages
• Missing answers
• Outdated documents
• Prompt-injection content
• Multiple sources
• Long context
• Irrelevant distractors

The teacher should demonstrate:

• Citation behaviour
• Evidence selection
• Uncertainty
• Refusal when evidence is absent
• Conflict resolution
• Separation of instructions from retrieved content

Without these cases, the student may simply learn to produce confident answers rather than grounded answers.

15. Distillation versus related optimisation techniques​

15.1 Distillation versus fine-tuning​

Fine-tuning means updating a model using a dataset.

Distillation describes where some training supervision comes from: a stronger teacher model.

Therefore:

Distillation is often implemented through fine-tuning.

All distillation may involve fine-tuning, but not all fine-tuning is distillation.

15.2 Distillation versus quantisation​

Quantisation reduces numerical precision.

Example:

The architecture and learned behaviour remain mostly the same, but weights occupy less memory.

Distillation trains a different model to imitate the teacher.

They can be combined:

15.3 Distillation versus pruning​

Pruning removes:

• Individual weights
• Attention heads
• Neurons
• Layers
• Channels

Distillation transfers behaviour into a student architecture.

A pruned model can be further distilled to recover lost performance.

15.4 Distillation versus RAG​

RAG injects knowledge at inference time.

Distillation changes model parameters during training.

Use RAG when:

• Information changes frequently
• Evidence must be cited
• Private documents are required
• Knowledge freshness matters

Use distillation when:

• Behaviour must become faster
• Output format must be consistent
• The task is repetitive
• The workflow is stable
• Inference cost must fall

Many production systems use both:

15.5 Distillation versus prompt caching​

Prompt caching avoids recomputing repeated context.

It does not transfer capabilities or reduce model parameter count.

Caching is best when the same long prefix is repeatedly submitted.

16. Why distillation fails​

16.1 The student is too small​

A 1B-parameter student may not have enough representational capacity to reproduce a frontier model across broad reasoning, coding, multilingual and agentic tasks.

Distillation cannot eliminate architectural capacity constraints.

16.2 The dataset is too narrow​

If training contains only successful, simple requests, the student will fail on:

• Ambiguity
• Edge cases
• Long context
• Tool failures
• Adversarial prompts
• New domains

16.3 Style imitation is mistaken for intelligence​

A student may learn phrases such as:

• “Let’s break this down.”
• “It is important to note…”
• “Here is a structured analysis…”

without acquiring the teacher’s underlying correctness.

Evaluation must measure outcomes, not stylistic similarity.

16.4 Teacher errors are amplified​

The student may inherit:

• Hallucinations
• Bias
• Incorrect policies
• Unsafe behaviour
• Over-refusal
• Poor calibration
• Outdated information

Teacher-generated data must be verified.

16.5 Synthetic-data collapse​

Repeated training on model-generated text can reduce:

• Linguistic diversity
• Rare knowledge
• Creativity
• Tail-distribution performance

Mitigations include:

• Human-written data
• Raw-domain data
• Multiple teachers
• Diverse sampling
• Deduplication
• Distribution monitoring

16.6 Exposure bias​

During SFT, the student observes the teacher’s correct previous tokens:

At inference, it conditions on its own generated history:

A small early mistake may therefore compound over a long response.

On-policy distillation, student-generated rollouts and sequence-level preference training can reduce this problem.

16.7 Training–deployment mismatch​

A student trained with perfect documents and successful tools may fail when production contains:

• Missing fields
• Timeouts
• Stale documents
• Malformed records
• Authentication failures
• Long conversations
• Unexpected languages

The training environment must approximate production.

16.8 Incorrect evaluation​

A distilled model may appear strong because:

• Training data overlaps with test data
• The judge prefers teacher-like wording
• Only easy examples were tested
• Safety failures were ignored
• Latency was measured on different hardware
• Costs excluded retrieval or infrastructure

17. Legal, contractual and governance considerations​

Knowledge distillation is a legitimate technique, but permission to use a particular provider’s outputs for training depends on:

• Provider terms
• Model licence
• Contractual agreement
• API access route
• Intended use
• Whether the resulting model competes with the provider
• Copyright and database rights
• Data-protection law
• Customer-data rights

Anthropic has stated that distillation is widely used legitimately, including by frontier laboratories to produce smaller models, while also describing unauthorised large-scale capability-extraction campaigns as distillation attacks.

Anthropic’s published Bedrock commercial terms have prohibited using the service to train competing AI models unless expressly approved. OpenAI’s current individual terms prohibit automatically extracting outputs and using outputs to develop models that compete with OpenAI. Terms differ across products and access routes, so the applicable current agreement must be reviewed before beginning a project.

An enterprise project should establish:

• Written permission for the intended distillation use
• Data-processing agreements
• Training-data lineage
• Output-retention rules
• Personal-data controls
• Intellectual-property review
• Security approval
• Model-risk assessment
• Audit logs
• Deletion procedures
• Third-party licence tracking

Owning an individual generated output does not necessarily mean that unrestricted automated collection or competitive model training is permitted.

18. A safe enterprise architecture​

19. Teacher fallback and model routing​

The student does not need to handle every request.

A production router can send easy cases to the student and difficult cases to the teacher.

Let:

Then:

The threshold (\gamma) controls the quality–cost trade-off.

More sophisticated routing considers:

For example:

Production failures routed to the teacher can become future distillation examples, creating an active-learning loop:

20. A practical project plan​

Stage 1: Discovery​

• Identify one narrow, high-volume workload.
• Establish teacher and student baselines.
• Define quality, latency, cost and safety targets.
• Obtain legal and security approval.

Stage 2: Pilot dataset​

• Collect 5,000–20,000 representative prompts.
• Remove or protect sensitive information.
• Generate teacher responses.
• Validate with deterministic rules and human sampling.
• Build a fixed evaluation set.

Stage 3: Initial training​

• Start with LoRA or QLoRA.
• Train on high-confidence examples.
• Mix in general instruction data.
• Compare against the base student.

Stage 4: Failure analysis​

Segment failures by:

• Intent
• Language
• Difficulty
• Context length
• Tool use
• Policy
• Safety
• Formatting

Generate targeted teacher examples for the largest failure clusters.

Stage 5: Preference and robustness training​

• Add preferred and rejected answer pairs.
• Include adversarial examples.
• Include tool failures.
• Include no-answer and escalation cases.

Stage 6: Production pilot​

• Deploy to a small traffic percentage.
• Keep teacher fallback.
• Track quality, cost, latency and complaints.
• Store approved failures for retraining.

Stage 7: Continuous improvement​

• Retrain when the distribution changes.
• Re-evaluate after policy updates.
• Monitor teacher and student version drift.
• Preserve previous checkpoints and dataset lineage.

21. Final perspective​

Knowledge distillation does not copy a large model parameter by parameter. It transfers selected behaviours and capabilities through training signals.

For Claude-like proprietary models, the most realistic authorised approach is usually:

For open-weight teachers, the pipeline can be richer:

The strongest production approach is rarely “copy every capability of the teacher.” It is usually:

1. Select a bounded workload.
2. Build a representative prompt distribution.
3. Generate authorised teacher supervision.
4. Verify and curate that supervision.
5. Train a correctly sized student.
6. Measure task outcomes rather than stylistic similarity.
7. Keep a teacher or human fallback for difficult cases.
8. Continuously learn from production failures.

Distillation works best when it is treated not as a one-time compression trick, but as a controlled capability-engineering lifecycle connecting data, evaluation, model training, governance and production monitoring.