- Distillation trains a small student model to mimic a large teacher on one task. The student inherits capability far above what its size would reach from scratch, because the teacher supplies the supervision.
- For models behind APIs, the practical path is response or sequence-level distillation: train the student on the teacher's generated outputs. Classical soft-target distillation is stronger but needs logit access you rarely have.
- The dataset is the entire project. Diverse, aggressively filtered teacher outputs make a fast specialist; unfiltered teacher output makes a confidently wrong one. Budget for the pipeline and for re-distillation when the teacher upgrades.
The idea: transfer, not train from scratch
Training a capable model from scratch is a research-scale undertaking: enormous data, enormous compute, and a long tail of decisions that only large labs are set up to make. Distillation sidesteps all of it. Instead of teaching a small model the task from raw data, you let an already-capable large model, the teacher, generate the supervision, and you train a small model, the student, to reproduce it. The student never has to discover the behavior on its own. It only has to imitate a competent demonstration of it, which is a far easier learning problem.
The result that makes the technique worth the trouble is that the student ends up performing well above what a model of its size would reach if trained on the same task from scratch. The teacher has already done the expensive work of learning a good input-to-output mapping, including the judgment calls and edge-case handling that a small model would struggle to induce from a limited dataset. Distillation transfers that learned function rather than rediscovering it. On a narrow, well-defined task the gap between a distilled small model and its teacher can be modest, while the gap between that same small model trained conventionally and the teacher is often large.
This is a transfer move, not a compression trick. You are not shrinking the teacher's weights; you are training a different, smaller architecture to occupy the same behavior on a slice of the input space. That framing matters, because it sets the honest expectation: the student inherits the teacher's competence where the teacher data covered, and nothing more. The teacher defines the ceiling and the dataset defines how much of that ceiling the student actually reaches.
Response, logit and sequence distillation
Distillation comes in flavors that differ by what signal the teacher exposes, and the distinction is not academic: it decides whether you can distill a given model at all. Classical knowledge distillation trains the student to match the teacher's full softened output probability distribution, the "soft targets." Instead of learning only that the correct next token is refund, the student learns that the teacher put most of its mass on refund but also considered return and credit plausible. That distribution carries far more information per example than a single hard label, which is why soft-target distillation is generally the most sample-efficient and highest-fidelity form.
The catch is that it requires access to the teacher's logits or probabilities across the vocabulary. For open-weight teachers you run yourself, that access is yours. For a frontier model behind a commercial API, it usually is not: you get generated text, not the distribution behind it. That constraint is what makes response-level and sequence-level distillation the common path for large language models. Here the teacher simply generates outputs, complete answers, structured extractions, labeled classifications, and the student is trained on those text outputs as ordinary supervised targets. You lose the soft-target signal, but you gain the ability to distill any model you can call.
| Flavor | Signal from teacher | Access needed | Typical use |
|---|---|---|---|
| Classical KD (soft targets) | Full probability distribution | Logits / probabilities | Open-weight teachers, max fidelity |
| Response-level | Final generated answer | Text output only | API teachers, most LLM work |
| Sequence-level | Full generated sequences | Text output only | Generation and reasoning tasks |
In practice, most enterprise distillation is response or sequence-level, because the best teachers are the ones you reach only through an API. It is worth knowing that this path leaves fidelity on the table: you are learning from samples of the teacher's behavior rather than its full uncertainty. When you control the teacher, soft targets are the stronger tool. When you do not, generated text is a perfectly workable substitute, and the dataset quality matters more than the flavor.
The dataset is the real work
The algorithms in distillation are the easy part. The dataset is the project, and it is where distillation efforts quietly succeed or fail. The student learns exactly what the teacher data shows it, so the entire game is making that data diverse, correct, and representative of the task you will actually see in production. A distilled model is a mirror of its training set with none of the teacher's ability to recover from gaps.
The generation step has to cover the task distribution, not just its easy center. Sample teacher outputs across the full range of inputs your system encounters: the common cases, the awkward phrasings, the adversarial inputs, the rare-but-costly edge cases. A student trained only on clean, typical examples looks excellent in a demo and falls apart on the long tail, because it never saw the long tail. Coverage of the input space is the single strongest predictor of whether the distilled model survives contact with real traffic.
Then comes the part teams skip: filtering, aggressively. Teachers are capable but not infallible. They hallucinate, they occasionally misformat, they sometimes reason to a wrong answer with total confidence. If you train the student on that output unfiltered, you do not average out the errors, you install them, and you install them with the teacher's confident tone attached. The output is a student that is wrong in the same places the teacher was wrong, but without any of the teacher's general capability to catch itself.
- Verify correctness where you can. Use ground-truth checks, execution (does the generated SQL run?), schema validation, or a second model as a critic to reject bad teacher outputs before they enter the set.
- Filter for quality, not just correctness. Drop lazy, truncated, or off-format answers; the student will faithfully reproduce mediocrity.
- Deliberately cover edge cases. Prompt the teacher for the hard inputs on purpose rather than hoping random sampling finds them.
- Deduplicate and balance. Repeated near-identical examples overweight one behavior and hide gaps elsewhere. Generation patterns are covered in Synthetic Data.
When distillation pays
Distillation is a cost-optimization move, and it earns its keep under a specific set of conditions. The ideal candidate is a task that is narrow, stable, and high-volume, where a frontier model already produces good results but is too slow or too expensive to serve at the scale you need. When those conditions hold, you distill the proven behavior into a small model that runs at a fraction of the cost and latency, and the economics of the workload change materially.
Each of the three conditions is load-bearing. Narrow, because the student has limited capacity and can only absorb a bounded slice of behavior; a broad, open-ended task exceeds what a small model can hold. Stable, because the distilled behavior is frozen at training time, and a task whose correct answers shift week to week will drift out from under a static student. High-volume, because distillation has real upfront cost, the teacher generation, the curation pipeline, the training and evaluation, and only high query volume amortizes that investment against the per-request savings.
does the task look like this?
|
narrow? -- no --> keep prompting the big model
| yes
stable? -- no --> keep prompting; too volatile
| yes
high-volume? - no -> keep prompting; won't amortize
| yes
v
DISTILL: frontier teacher --> small student
cheaper + faster to serve at scale
The inverse is just as important to state plainly. Do not distill a broad task, an experimental task whose specification is still moving, or a low-volume task where the frontier model's per-call cost is already a rounding error. In those cases the pipeline cost dominates and you have spent engineering effort to make something slightly cheaper that was never the expensive part. Distillation is what you reach for after a workload has proven itself with a prompted frontier model and grown into a cost line worth optimizing, not before.
Distillation vs fine-tuning vs prompting
Distillation sits alongside prompting and fine-tuning as the three levers of adaptation, and they answer different questions. Prompting the large model is the fastest thing to try and should almost always come first: no training, no dataset, no infrastructure, just instructions and examples in the context window. It proves whether the task is solvable at all and gives you a working baseline in an afternoon. Its cost is that you keep paying frontier per-token prices and frontier latency on every call, forever.
Fine-tuning adapts a model's behavior by updating its weights on your examples: it teaches format, tone, and task reflexes that prompting struggles to pin down, as detailed in the fine-tuning deep-dive. Distillation is the lever aimed specifically at cost and latency: its distinguishing purpose is to buy the economics of a small model while keeping much of the large model's quality on a narrow task. The other techniques change what a model does; distillation changes how expensive it is to do it.
| Lever | What it buys | Main cost | Reach for it when |
|---|---|---|---|
| Prompting | Fastest path to a working result | Ongoing frontier price and latency | Validating a task; low volume |
| Fine-tuning | Behavior: format, tone, reflexes | Dataset discipline; a training run | Prompting can't pin the behavior |
| Distillation | Small-model cost and latency | Teacher generation + data pipeline | Proven, narrow, high-volume task |
These are not mutually exclusive, and the strongest pipelines combine them. A common pattern is to distill a frontier teacher's behavior into a small base model to establish the core capability, then apply a light fine-tune, often LoRA, to sharpen format or align tone to your brand. The distillation transfers the hard-won task competence; the LoRA pass polishes the manner cheaply. Read the sequence as a funnel: prompt to prove the task, distill to make it cheap at scale, fine-tune to finish it.
The risks: coverage, drift and dependence
The defining limitation of a distilled model is that it is only as good as the slice of behavior the teacher data covered. Within that slice it can be excellent; outside it, it has nothing to fall back on. The frontier teacher had broad general capability to reason its way through an unfamiliar input. The small student does not, and it will produce a fluent, confident answer for inputs it never learned to handle. Coverage gaps in your dataset become silent failure regions in production.
This makes distribution shift the acute operational risk. A distilled model is a snapshot of a task frozen at training time. When the real distribution moves, new product names, new customer phrasings, a new category of request, the student degrades precisely because it cannot generalize the way its teacher could. What looked like a robust specialist becomes a brittle one, and the failure is easy to miss because the model stays fluent while it becomes wrong. Continuous evaluation against fresh production traffic is the only reliable detector.
Finally there is the dependence the technique creates on the teacher itself. The moment your teacher upgrades to a materially better version, your distilled student is behind: it encodes the old model's behavior, and it will not improve on its own. Staying current means periodically regenerating data and re-distilling, the "re-distill treadmill." That is a recurring operational cost, not a one-time build, and a distillation strategy that does not budget for it will quietly fall behind the frontier while everyone assumes the model is done.
The architect view
Treat distillation as what it is: a cost-optimization technique for a proven, narrow workload at scale, not a default and not a research project. It belongs late in a workload's life, after prompting has validated the task and volume has grown into a cost line worth attacking. Reaching for it earlier inverts the effort: you spend a data pipeline's worth of engineering to optimize something that was never expensive, and you freeze a task specification that was still moving.
The budget line that teams underestimate is the data pipeline, not the training. Distillation is a data-engineering effort wearing a machine-learning label. Fund the teacher generation, the correctness filtering, the coverage of edge cases, and the deduplication as the core of the project, because that is where student quality is actually decided. Fund it once for the build, and fund it again on a schedule, because teacher upgrades and distribution shift both force periodic re-distillation. A distilled model is a maintained asset with a standing cost, not a delivered artifact.
- Gate on the three conditions. Narrow, stable, high-volume, and a frontier model that already works. Miss any one and keep prompting.
- Evaluate the student against the teacher continuously. Track the quality gap on live traffic, not a frozen test set, so drift shows up as a trend before it shows up as an incident. The harness pattern lives in the Eval Harness.
- Serve it behind the same gateway as the rest of your model traffic, so swapping teacher, student, or the prompted fallback is configuration rather than a redeployment.
- Clear the terms of service first. Confirm the teacher's outputs may be used to train your model before you commit to a teacher.
Done with that discipline, distillation is one of the more durable wins in the adaptation toolkit: a small model that beats a prompted large one on your task, at a cost and latency that make a high-volume workload viable. The illustrative economics, a distilled small model serving a fraction of the teacher's per-call cost, only hold when the dataset earns them and the re-distillation stays on schedule.