Model Distillation
The idea was set out by Hinton, Vinyals and Dean in Distilling the Knowledge in a Neural Network (2015), which showed that the knowledge in a large model, or an ensemble of models, could be compressed into a single smaller model that is easier to deploy. The student learns from the teacher's outputs instead of only from human labels.
In LLM work, distillation usually means running a strong model over many real inputs from your task, checking the outputs, and fine-tuning a smaller model on those pairs, often with LoRA. The student gets good at that one task and can be much cheaper to run. Check the teacher model's terms of use first, since some providers restrict using their outputs to train other models. Hold back a golden dataset to confirm the student matches the teacher, and consider model routing to send the cases it struggles with back to the larger model.
Example
# Distillation in four steps
1. Collect a few thousand real inputs for one task (for example, ticket triage).
2. Run the teacher model on each input with your best prompt. Save input and output pairs.
3. Have people review a sample, and drop or fix bad outputs.
4. Fine-tune a small model on the pairs, then compare it with the teacher on a held-out golden dataset.When to use it
- A narrow, high-volume task where a large model works but costs too much
- You need lower latency or to run the model on your own hardware
- The task is stable enough that a trained model won't go stale quickly
When not to use it
- Low volume, where paying for the large model is cheaper than training
- Broad, changing tasks a small model can't cover
- The teacher's terms don't allow it

