ModelRefs / Knowledge Distillation — AI Glossary
Knowledge Distillation — AI Glossary
Training a smaller student model to mimic the outputs of a larger teacher model, transferring capabilities at reduced parameter count.
Overview
Distillation (Hinton et al. 2015) trains the student on teacher soft-probability distributions (dark knowledge) rather than hard labels. For LLMs: DistilBERT (40% smaller, 97% BERT performance), TinyLlama, and Phi series use distillation. Teacher-generated synthetic data for instruction tuning is a soft-distillation variant.
Reference details
| Topic | training |
|---|---|
| Also known as | distillation, student-teacher training |
| Last reviewed | 2026-06-24 |
Related terms
Example: What the soft targets carry that a label does not
A hard label says the answer is “cat”. The teacher's distribution says cat 0.70, dog 0.25, fox 0.04, car 0.01. The extra information is in the ordering of the wrong answers: the teacher has encoded that this image is dog-like and not at all car-like, which is a similarity structure the one-hot label deletes entirely. The student trained on the distribution learns that structure and reaches accuracy it could not reach from labels alone with the same parameters. Raising the softmax temperature spreads the distribution further and makes those small differences count more.
Commonly confused with
Classical distillation trains on the teacher's output distribution and needs logit access. What is usually called distillation for LLMs today is different — generating text from a strong model and fine-tuning on it, which is supervised fine-tuning on synthetic data. It works, but it transfers only what the samples contain, and provider terms often restrict training competing models on their outputs.
When to use it
Reach for it when:
- Compressing a capable model into one you can afford to serve at volume
- Specialising a small model to a narrow task where the teacher is overqualified
- Where inference cost dominates and a few points of quality are an acceptable trade
Reach for something else when:
- Expecting the student to generalise beyond the distribution it was distilled on
- Without checking the teacher's terms of service for output-training restrictions
- As a substitute for better data: distilling a flawed teacher reproduces the flaws
Primary source
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Knowledge Distillation — AI Glossary.
Frequently asked questions
What is Knowledge Distillation?
Training a smaller student model to mimic the outputs of a larger teacher model, transferring capabilities at reduced parameter count.
Is Knowledge Distillation the same as distillation?
Yes — distillation, student-teacher training are common aliases for Knowledge Distillation.
What concepts are related to Knowledge Distillation?
Closely related concepts include pruning, model distillation, model soup.