NOTES / FINE-TUNING AND ADAPTATION

Distillation

Training a small student to imitate a large teacher: classic soft-target distillation, text-based distillation from API models, and the legal and quality traps.

UPDATED 18 NOV 2026#fine-tuning#distillation#synthetic-data#student-teacher

The idea

A large model can be expensive to run. Distillation trains a smaller student to reproduce the behaviour of a larger teacher, aiming to keep most of the quality at lower cost and latency (Latency and cost).

Classic distillation

Hinton and colleagues proposed using the teacher’s class probabilities as soft targets for the student. When those soft targets have high entropy they carry much more information per training case than hard labels, which lets the student learn how the teacher generalises1. A temperature parameter softens the probabilities during training1.

DistilBERT applied this during pre-training and reported a model 40 percent smaller and 60 percent faster that kept about 97 percent of BERT’s language-understanding performance on their benchmark2. Those figures are the authors’ and apply to that model and task set.

This approach needs access to the teacher’s output probabilities, so it fits open-weight teachers.

Text-based distillation

With an API model you usually get text, not probabilities. The common recipe is to have the teacher generate training examples and then run supervised fine-tuning on the student. Self-Instruct showed how a model can bootstrap instructions and responses from its own generations, starting from a small human-written seed set4. Stanford’s Alpaca applied this: it started with 175 human-written seed pairs, used text-davinci-003 to generate 52,000 instruction-response examples at a reported cost of under $500, and tuned LLaMA 7B on them3.

Diagram: with open weights the student learns from the teacher's token probabilities; with an API model it learns from the teacher's generated text

Diagram: my own.

A practical pipeline

Diagram: real prompts from your task, teacher answers, filter and verify, train the student, compare to the teacher

Diagram: my own.

  1. Use real prompts from your task, not generic ones, so the student sees the distribution it will face.
  2. Generate answers from the teacher, possibly several per prompt.
  3. Filter and verify. Drop wrong or low-quality outputs with checkers, rules or a judge model (LLM-as-judge). The student inherits the teacher’s mistakes.
  4. Train with SFT, often with adapters (LoRA and QLoRA).
  5. Compare student and teacher on a held-out eval set, per category, and set a quality floor before switching traffic.

Traps

  • Provider terms. Alpaca’s authors noted that the data came from a model whose terms prohibit developing competing models, and that LLaMA’s licence was non-commercial, which is why they restricted use to research3. A commentary reports that major providers include similar conditions6. Read the current terms of any provider before using its outputs as training data. I am not a lawyer; ask one if this matters to you.
  • Recursive degradation. Shumailov et al. found that models trained on model-generated data lose the tails of the original distribution, and that access to real data is needed to avoid this5. Mix in real, human-verified examples and avoid chaining students on students.
  • Narrow gains. A student tuned on one task may do worse elsewhere.
  • Overstated parity. Averages hide the hard cases. Look at slices.

When it pays

Distillation pays when volume is high, the task is narrow, and a quality floor can be measured. It is a late step on the ladder in When to fine-tune, not a first one.

Sources

  1. [1]Distilling the Knowledge in a Neural Network — Hinton, Vinyals, Dean · 2015PARAPHRASED
  2. [2]DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter — Sanh, Debut, Chaumond, Wolf · 2019PARAPHRASED
  3. [3]Alpaca: A Strong, Replicable Instruction-Following Model — Stanford CRFM · 2023PARAPHRASED
  4. [4]Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · 2023PARAPHRASED
  5. [5]The Curse of Recursion: Training on Generated Data Makes Models Forget — Shumailov et al. · 2023PARAPHRASED
  6. [6]When Does Distilling GPT Become Theft? — Legal Layer (Substack)COMMENTARY ON PROVIDER TERMS; NOT LEGAL ADVICE

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.