What is Evaluation?
LLM evaluation (evals), told as a story about an examiner grading every parrot on the same workbook.
The parrot got swapped. Nobody knows if it got better or worse.A new parrot, a new note, a new drill — whatever you change, the same question comes up. Is it better? Feelings will not answer that.
The answer wobbles each time, and every guest has a different bar.Because of the wobble dial, even the same question gets different answers. And what counts as good differs from person to person. So a glance or two cannot compare.
Evaluation is the examiner grading every parrot on the same workbook.The workbook is 100 questions drawn from our own job. Same questions, same grading — that is what makes the numbers comparable.
With a scorecard you compare by numbers, not feelings.Questions with one right answer get graded by a machine. Ones without — like a summary — get scored by a judge parrot or a person. Add it all up for the total.
Beware the parrot that only aces the workbook, and remember the world test is not our test.If the workbook leaks, the parrot memorizes the answers (benchmark overfitting). Top of the world test is not top of our job. Whenever you drill it, choose a parrot, or just look after it, re-test on your own workbook.
Evaluation = the examiner grading every parrot on the same workbook to make a scorecard. A machine grades questions with one answer, a judge parrot or a person grades the rest, and you re-test at every change. Beware the parrot that only aces the workbook.
Repeatedly measuring the quality of a model, prompt, or system on a fixed evaluation set. Questions with ground truth are auto-scored as accuracy or pass rate; open-ended ones are scored by LLM-as-judge or human raters. Re-running on every change (regression testing) is the core practice, and public benchmarks cannot stand in for your own task because of contamination and overfitting.
When grown-ups say it
- Eval
- The examiner's scorecard. Numbers instead of feelings. Run again at every change.
- Evaluation set
- Our workbook. Questions and expected answers drawn from our own job. A hundred is enough to start.
- Benchmark
- The world test. A public workbook everyone takes. Top of the world test is not top of our job.
- LLM-as-judge
- Another parrot that grades answers. Scores open-ended answers like summaries and tone. Being a parrot, it too is sometimes wrong.
- Accuracy · pass rate
- How many it got right. For questions with one answer, a machine counts. 92 of 100 right is 92.
- Regression test
- Re-test at every change. If the score drops, something got worse — roll back.
- Human evaluation
- A person grades by hand. Most accurate, but slow and costly. Used to check that the judge parrot grades well.
- Routing
- Which parrot gets the job. Pick a parrot per task by looking at the scorecard. → which parrot gets the job