Explain like I'm five · AI

What is Distillation?

Knowledge distillation, told as a story about collecting a big parrot's answers and teaching a small parrot to copy them.

…one moment… the answer is… Parisgreat… but too slow and pricey for every daya guest who asks every daythe big parrot is great — but slow, and every answer costs a heap of beans

The big parrot is great. But it is slow, and every answer costs a heap of beans.For a job that asks thousands of times a day, that adds up. A small parrot would be nice — but a small one is not as good.

the whole mountain, from scratchsmall parrotSCHOOL BILLa million sacks of beansseveral monthsa hundred trainerstrainerraising a small parrot on books again costs too much — school is a once-only thing

Raising a small parrot on the whole mountain of books again costs far too much.The reading school (pretraining) costs a million sacks of beans. You cannot reopen it for one job.

the capital of France is Paristeacher parrotANSWERit is ParisANSWERit is ParisANSWERit is ParisANSWERit is Paristhousands of answer notestrainerit is Paris!student parrotdistillation: collect heaps of the big parrot's answers, and teach the small parrot to copy them

Distillation collects heaps of the big parrot's answers and teaches the small parrot to copy them.The big parrot is the teacher, the small one the student. It learns from the teacher's answer notes instead of books — far cheaper and faster than school.

Teacher answers as a workbookcollect thousands of Q and A
Only our kind of problemsour questions, not any questions
The small one does that job wellstill short on other jobs
Check with a testcompare with the teacher's score
1 the big parrot answersANSWERQ capital of France?A ParisANSWERQ capital of France?A ParisANSWERQ capital of France?A Paris2 notes ×3,000DRILLcopy it3 drill the small parrotit is Paris!4 answers alikestep 3 is fine-tuning — a few thousand notes, not a mountain of books

Thousands of the big parrot's answer notes drill the small parrot.Step 3 is fine-tuning. Because the teacher's notes were not written by people, they are called synthetic data.

teacherstudentTESTteacher 95student 92the student gets close, but rarely passes the teachera book from 19871987!it copies the teacher's mistakes tooteaching from someone else's parrot = check the terms firstsmall, but good enough for our job — that is the goal of distillation

The student rarely passes the teacher, and copies the teacher's mistakes too.So always confirm with a test. Teaching from someone else's parrot raises terms-of-service issues. Sending only the easy jobs to the small parrot is routing; shrinking it another way is quantization.

In one breath

Distillation = collect heaps of a big parrot's answers and teach a small parrot to copy them. Far cheaper than school, and good enough for our job. The student rarely passes the teacher and copies its mistakes.

Training a small student model on the outputs of a large teacher model, keeping task performance while cutting cost and latency. The teacher outputs are synthetic data, and the student training is fine-tuning (SFT). Training on another vendor's model outputs may violate its terms of service.

When grown-ups say it

Knowledge distillation
Teaching the small parrot to copy. The big parrot's answers train the small one.
Teacher · student model
The teacher parrot and the student parrot. The teacher is big and good; the student is small and fast.
Synthetic data
Answer notes a parrot wrote. A workbook made by a parrot, not people. → teaching a parrot with a parrot's book
Small language model (SLM)
The small parrot. Fits on a phone or laptop. Good at one job.
Fine-tuning
The drill on answer notes. How the student learns. → the short drill
Versus quantization
Teach a new bird vs lighten the same bird. Distillation raises another parrot; quantization rewrites the same parrot's numbers coarsely. → counting beans coarsely
Routing
Which parrot gets the job. Easy jobs to the student, hard ones to the teacher. → which parrot gets the job
Evaluation
The teacher-and-student test. Confirm with scores that the student is good enough. → the examiner's scorecard
Next story
Counting Beans Coarsely →
Want another term as a picture book? Request →