Model Distillation: How Small Models Learn from Big Ones

A 400-billion-parameter model is expensive to serve: it needs multiple GPUs just to hold the weights, and every token costs real money and real latency. A 7-billion-parameter model is cheap to serve almost anywhere, including on a single consumer GPU. The gap in capability between them used to be the whole story. Distillation is the technique that lets you shrink that gap for a specific task, by training the small model not from raw data, but from the large model’s own behavior.

The core idea is old — it predates the current wave of LLMs by nearly a decade — but it’s become one of the most economically important techniques in the field, because it’s the mechanism behind almost every small model that punches above its parameter count.

Why Not Just Train Small From Scratch

If you train a small model directly on the same raw dataset as a large one, you get a small model that’s simply less capable, because it has less capacity to represent the patterns in that data. Distillation instead trains the small model — the student — to reproduce the outputs of an already-trained large model — the teacher. The insight is that a teacher’s outputs carry far more information than the raw labels they were trained on. A teacher doesn’t just say “this is a cat”; it assigns some probability to “dog”, a little to “fox”, and near-zero to “airplane”. That distribution is a signal in its own right, and it’s the signal a hard label throws away.

Soft Labels: The Information a Hard Label Discards

In classification, a hard label is a one-hot vector: 100% “cat”, 0% everything else. A teacher model’s raw output, before you take the argmax, is a full probability distribution — a soft label. Training the student to match that full distribution, rather than just the single correct answer, teaches it the teacher’s implicit sense of which mistakes are reasonable.

For language models the same idea applies at the token level: instead of training the student purely on the next correct token (hard label), you train it to match the teacher’s full probability distribution over the vocabulary at each position.

Temperature: Turning Up the Softness

A model’s raw output logits, passed through softmax, tend to be extremely peaked — the correct token often gets a probability near 1.0 with everything else close to zero, which looks almost as uninformative as a hard label. Distillation applies a temperature parameter to soften this before computing the loss:

import torch
import torch.nn.functional as F

def soft_targets(logits, temperature):
    return F.softmax(logits / temperature, dim=-1)

def distillation_loss(student_logits, teacher_logits, temperature=2.0, alpha=0.5, hard_labels=None):
    soft_teacher = soft_targets(teacher_logits, temperature)
    soft_student = F.log_softmax(student_logits / temperature, dim=-1)
    # KL divergence between softened distributions, scaled back up
    kd_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (temperature ** 2)
    if hard_labels is None:
        return kd_loss
    ce_loss = F.cross_entropy(student_logits, hard_labels)
    return alpha * kd_loss + (1 - alpha) * ce_loss

Higher temperature flattens the distribution further, exposing more of the teacher’s relative ranking among the wrong answers. The temperature ** 2 rescaling exists because softening shrinks the gradient magnitude; without it, a high-temperature loss barely moves the student’s weights. Most recipes blend the softened KL term with a normal hard-label cross-entropy term (alpha above), so the student still anchors to ground truth and doesn’t drift purely toward mimicking the teacher’s quirks.

Logit Distillation vs. Data Distillation

There are two distinct families of technique, and they get conflated constantly:

Logit distillation requires white-box access to the teacher: you need its raw logits (or at minimum its full output distribution) for every training example, computed in the same forward pass, so you can compute a KL-divergence loss against them directly. This is the classical approach and it’s what the code above implements. It only works when you can actually run the teacher yourself, or when a provider exposes logprobs.

Data distillation (sometimes called “black-box distillation” or, informally, “synthetic data distillation”) doesn’t need logits at all. You use the teacher purely as a generator: prompt it to produce completions, explanations, or labeled examples, and then fine-tune the student on that generated text with ordinary supervised learning.

# Data distillation: use a large hosted model purely as a labeler/generator.
# No logits required — just its text output.
def generate_training_example(client, teacher_model, question):
    response = client.chat(
        model=teacher_model,
        messages=[{
            "role": "user",
            "content": f"Answer step by step, then give a final answer.\n\n{question}",
        }],
    )
    return {"prompt": question, "completion": response.content}

dataset = [generate_training_example(client, "teacher-large", q) for q in questions]
# dataset now becomes ordinary supervised fine-tuning data for the student

This is the practical route for almost anyone distilling from a closed commercial API, since those APIs generally don’t expose raw logits (and even when they expose logprobs, it’s usually only for the top few tokens, not the full vocabulary). It’s also the reason “distillation” and “fine-tuning on synthetic data from a bigger model” have become nearly synonymous in casual usage, even though logit distillation is the technically stricter, more information-dense version of the idea.

Property	Logit distillation	Data distillation
Requires teacher logits	Yes	No
Works against a closed API	Rarely	Yes
Information per example	Full output distribution	Single generated sample
Typical use case	Compressing your own trained model	Bootstrapping a small model from a hosted large one
Compute cost	One forward pass per example, both models	One generation call per example, teacher only

Why a Distilled Model Can Outperform Its Teacher on Narrow Tasks

This sounds counterintuitive — how can the student beat the model it learned from? — but it happens regularly, and the explanation is about specialization, not magic.

A large general-purpose teacher spends its capacity across an enormous range of tasks: languages it rarely gets asked about, domains far outside your use case, formatting conventions you’ll never need. A student distilled specifically on your task distribution puts its entire (much smaller) capacity budget toward that one distribution. If you distill a large model’s behavior on, say, structured invoice extraction, the small student never has to “remember” how to write poetry or debug Rust; every parameter it has is doing useful work for the one job you care about.

There’s a second, more subtle effect: distillation on synthetic data filtered for correctness (only keeping teacher outputs that pass a verifier, a unit test, or a rule-based check) removes a chunk of the teacher’s own mistakes from the training signal before the student ever sees them. The student ends up trained on a curated subset of the teacher’s best behavior rather than its average behavior, which can push narrow-task accuracy above the teacher’s own unfiltered average on that task.

A Self-Hosted Recipe

For a small team without teacher-logit access, the practical path looks like this: call a hosted large model to generate labeled examples for your specific task, filter them for quality, and fine-tune an open-weights small model on the result.

# distill_config.yaml — data-distillation fine-tune, illustrative
base_model: "small-open-7b"
dataset: "./data/teacher_generated_invoices.jsonl"
method: "lora"
lora_rank: 16
epochs: 3
learning_rate: 2e-4
filter:
  min_confidence: 0.8      # drop teacher outputs below this self-reported confidence
  require_valid_json: true # drop any example that fails schema validation

Running the resulting student model on a single GPU, or even CPU for low-traffic endpoints, at a fraction of the teacher’s serving cost, is usually the entire point of the exercise.

Pitfalls

Inherited bias. A student trained purely on a teacher’s outputs inherits every systematic bias and blind spot the teacher has, with no independent correction, since the student never sees raw ground truth for anything it wasn’t explicitly given.

Mode collapse on narrow synthetic data. If your generation prompts aren’t diverse enough, the teacher produces repetitive phrasing and the student overfits to that narrow style, losing robustness on inputs that look even slightly different from the training distribution.

Licensing terms. Several commercial model providers’ terms of service restrict using their outputs to train a competing model. This is a legal question, not a technical one, but it’s worth checking before you build a pipeline around it.

Key Takeaways

• Distillation trains a small “student” model to reproduce a large “teacher” model’s behavior, rather than training from raw labels alone.
• Soft labels (the teacher’s full output distribution) carry more information than hard labels, which is the entire point of logit distillation.
• Temperature scaling flattens the teacher’s distribution so the student can learn from its ranking among wrong answers, not just the top pick.
• Logit distillation needs white-box access to the teacher; data distillation only needs its text output, which is why it’s the common path against closed APIs.
• A distilled model can outperform its teacher on a narrow task because it puts all its capacity toward that task and can be trained on filtered, higher-quality examples.
• Distillation inherits the teacher’s biases and blind spots wholesale; it is not an independent quality check.
• Check licensing terms before distilling from a commercial provider’s outputs.