What is Synthetic Data?
Synthetic data, told as a story about having the big parrot write piles of example cards to use as the small parrot's drill material.
The drill needs 500 of our examples — far too slow to write by hand.The short drill runs on example cards. It needs a few hundred; at an hour each, that is over a month.
Real examples are expensive — and carry secret beans.Hand-written cards cost money one by one. Using real guest stories as they are puts secret beans — phone numbers, cards — into the drill material.
Synthetic data is having the big parrot write piles of examples, then teaching the small one with them.People write only a few seed cards; the big parrot fills in the rest by imitation. A close cousin of teaching the small parrot to imitate.
Ten seeds, 500 cards, a person keeps 420, and the drill runs on those.The big parrot slips in plausible lies too. So a person filters, and the examiner's score sheet measures how good the kept cards are.
Fed only parrot-written books, the parrot gets stranger each generation. Always mix in real books.A parrot-written book imitates real ones; imitations of imitations lose the rare things first (model collapse). So synthetic data assists the drill — it never replaces the mountain of books.
Synthetic data = give a few hand-written seed cards, have the big parrot write piles of examples, let a person filter them, and teach the small parrot with the rest. Cheap, plentiful, free of secret beans — but a parrot fed only parrot books gets stranger.
Generating training and evaluation examples with a large model. Seed data sets the direction, diversity is enforced, and quality filtering (deduplication, error removal) precedes fine-tuning or distillation. It can produce data free of personal information, but training repeatedly on model output alone erases the tails of the distribution (model collapse), so real data must be mixed in.
When grown-ups say it
- Synthetic data
- Example cards the parrot wrote. Drill material made by the big parrot's imitation, not by people.
- Data augmentation
- Stretching the cards you have. Change the tone, the order, the names — one card becomes many.
- Model collapse
- Imitations of imitations going strange. Fed only parrot books, the rare things vanish first. Mix in real books.
- Seed data
- The first few cards, by hand. The standard the big parrot imitates. Bad seeds make 500 bad cards.
- Quality filtering
- A person sorting the cards. Wrong and duplicate cards go out. How good the rest are: → the examiner's score sheet
- Distillation
- The big parrot's ways, taught to the small one. A cousin of synthetic data. → teaching the small parrot to imitate
- Fine-tuning
- The short drill on example cards. Where the synthetic cards get used. → the short drill
- Privacy
- Made without secret beans. Made-up names and numbers keep real guest data out. → the secret beans we gave the parrot