What is Pre-training?
Pre-training, told as a story about a parrot school that hides the book and makes the parrot guess the next bean, a trillion times over.
A just-hatched parrot can't continue a single bean.So where does a talking parrot come from? We said it read a mountain of books — but how does that reading work?
Reading it all takes a big school, months, and a lot of money.Thousands of calculators run day and night. The power meter maxes out and the calendar flips for months. That is why only a few places raise big parrots.
Pre-training is a school that hides the book and makes the parrot guess the next bean — nobody teaches the answers.The book itself is the answer key. When it is wrong, its inner dials get a small nudge and it moves to the next line. Do that a trillion times and you get the mountain-of-books parrot. A drill and manners lessons follow.
More books and a bigger bird guess better. And school is not the end.Grow the books, the bird and the computing together and the parrot keeps improving — in one line: bigger and more means better. After school, a short drill fixes its habits and manners lessons fix its attitude.
Fresh out of school, the parrot continues well but knows no manners and nothing recent.Doing what it is told comes from manners lessons. What happened after it closed the books is in the day it closed the books — that gap is filled by the librarian, not the school.
Pre-training = the parrot school hides the book and has it guess the next bean a trillion times. Nobody teaches answers — the book is the answer key. Wrong guesses nudge the dials. More books and a bigger bird do better, but a drill and manners lessons must follow.
The stage where a model learns next-token prediction on a huge text corpus by self-supervision. Gradient descent nudges the parameters to reduce the loss — the gap between prediction and the real token. It takes thousands of GPUs, weeks to months and a lot of money; scaling laws say performance rises predictably when data, parameters and compute grow together. Fine-tuning and alignment (RLHF and the like) follow.
When grown-ups say it
- Pre-training
- The school of reading. Guess the next bean, a trillion times. Talking parrots come from here.
- Self-supervised learning
- Cover the book and guess. No one writes the answers. The next line of the book is the answer.
- Training data · Corpus
- The school's mountain of books. More and cleaner books make a better parrot. Bad books mixed in teach bad words too.
- Loss
- How far off it was. The number on the score note. The school only turns the dials to make it smaller.
- Scaling laws
- Bigger and more means better. Grow books, bird and computing together and the gain can be predicted ahead.
- GPUs · Training cost
- Thousands of calculators and the power meter. Months and a lot of money. That is why big parrot schools are few.
- Fine-tuning
- The short drill after school. Fixes habits and adds a specialty. → the short drill
- Knowledge cutoff
- The day it closed the books. Nothing after that day was in school. → the day it closed the books