What is Alignment?
Alignment, told as a story about manners lessons that teach a fresh-from-school parrot to answer the way people want — helpfully, honestly, harmlessly.
Fresh out of school, the parrot answers bad questions freely and rambles at simple requests.It continues sentences beautifully. But it has no idea what it should do, or should not.
The books hold everything, and school only taught the next bean.The school of reading taught it to continue good words and bad words alike. Someone has to teach it, separately, which answers are good.
Alignment is manners lessons so the parrot answers the way people want.Helpful, honest, harmless. Good answers earn treats, and rules written on a scroll are used for grading too. The method resembles the short drill, but what it teaches is attitude, not knowledge.
The scorer learns what people picked, then teaches the parrot. Tens of thousands of times.People can't read every answer. So a scorer learns how to hand out treats, and that scorer teaches the parrot instead. Grading by the rules scroll needs even fewer human hands.
Manners polish the surface. Fences where it leaks, and just enough caution.Ask sideways and something still leaks — so a fence is needed as well. Too much caution and you get a useless parrot that can't even peel an apple. Whether the balance is right is measured by the examiner.
Alignment = manners lessons so the parrot fresh from school answers the way people want (helpful, honest, harmless). Treats go to the better of two answers, and a rules scroll grades too. It polishes the surface, so a fence is still needed — and too much caution makes it useless.
The stage that adjusts a pre-trained model to act in line with human intent and values. The classic method is RLHF: a reward model is trained on human preference data between pairs of answers, then the policy (model) is reinforced to maximize that reward. Constitutional AI lets the model grade itself against written principles. Over-refusal is a side effect, and jailbreaks still call for separate guardrails and evaluation.
When grown-ups say it
- Alignment
- Manners lessons for the parrot. Teaches the attitude: helpful, honest, harmless.
- RLHF
- Treats for good answers. Treats go to the answer people picked, and the parrot's dials turn that way.
- Reward model
- A scorer that learned to give treats. People can't read every answer, so the scorer hands out treats in their place.
- Preference data
- Marks on the better of two answers. B over A — tens of thousands of such marks teach the scorer.
- Constitutional AI
- Grading by the rules scroll. Rules are written down and the parrot fixes its own answers against them.
- Over-refusal
- The parrot that can't peel an apple. Teach too much caution and it refuses perfectly fine requests.
- Guardrail
- The fence. Manners polish the surface, so leaks are blocked by a fence. → the fence
- Fine-tuning
- The short drill. Similar method, but the drill teaches knowledge and tone; manners teach attitude. → the short drill