What is Fine-tuning?
Fine-tuning, told as a story about a short drill for a well-read parrot, using a few hundred of our own examples.
The parrot doesn't know our house's tone or words. It uses settle differently than we do.The mountain of books has the whole world's words — but not the words and sheet shapes only our house uses.
Either write a long note every time, or drill it once. But a drill is not re-reading the books.A long note costs many beans every time. A drill takes once — not re-reading the mountain of books, but showing a well-read bird a few hundred examples.
Fine-tuning is a short drill for a well-read parrot, using a few hundred of our examples.The ingredient is examples — hundreds to thousands. It learns tone and sheet shapes well. New knowledge doesn't stick — the librarian putting pages on the tray (RAG) works better for that. After the drill comes a test.
Drills come in kinds: copying examples, fixing with treats, changing only a small sticker.Copying is the basic one. Treats teach it to pick the better of two answers. The small sticker doesn't change the whole bird, so it's cheap and fast.
Too much drill and it memorizes the examples and forgets the rest. And the drill is the last card to play.Try the note first; if that falls short, the librarian puts pages on the tray; only then the drill. Even a drill can't fully cure making things up — that's the plausible parrot story.
Fine-tuning = a short drill for a well-read parrot using a few hundred of our examples. It learns tone and format well; new knowledge is better left to the librarian (RAG). Too much and it only memorizes, so test after the drill. Order: note, librarian, drill.
Additional training of a pre-trained model on a small labeled set (hundreds to thousands of examples) to match tone, format and task behavior. SFT (supervised fine-tuning) is the base; RLHF rewards preferred answers from pairwise comparisons; LoRA trains only small adapters, making it cheap and fast. Validate on an evaluation set to catch overfitting and catastrophic forgetting, and try prompt, then RAG, then fine-tuning, in that order.
When grown-ups say it
- Fine-tuning
- The short drill. Teaches a well-read parrot our manners and formats using our examples.
- Supervised fine-tuning
- Copying the examples. Show question-and-answer cards and have it say the same. The most basic drill.
- Reinforcement learning from human feedback
- Fixing with treats. A treat for whichever of two answers people liked more. Politeness and safe answers come from here.
- Low-rank adaptation
- The small sticker drill. Changes a small sticker instead of the whole bird. Cheap, fast, and peel it off to get the original bird back.
- Overfitting
- Memorizing only the examples. Too much drill: perfect on the example cards, lost on new questions.
- Catastrophic forgetting
- Forgetting the rest while drilling. Drill only settlement sheets and it can't talk about weather anymore. Mix examples and keep the drill short.
- Evaluation set
- The test after the drill. The examiner measures with examples not used in the drill. Testing with drill cards is cheating.
- Prompt vs fine-tuning
- A note every time vs a drill once. The note comes first — cheap and instant. → the trainer's note