Explain like I'm five · AI

What is LLMOps?

LLMOps, told as a story about caring daily for a parrot that serves guests — noting note versions, logging chats, measuring beans and time, running tests, ringing the bell.

it was fine yesterdaysettle… like settling downsince when? why? no idea…LOG(empty)one day the answers went odd — nobody knows since when, or why

The parrot serves guests now. One day its answers went odd — and nobody knows since when, or why.It worked fine on launch day. But life after launch is the real thing — without a record you can't find when the plausible wrong answers started leaking.

things keep changing, nobody writes it downnew parrot versionNOTE v?who changed it?the note changedthe shelves changedLOG(none)no recordwhat is settle?month-end tallysettling downsame question, different answeranswers differ every time, so a breakdown is hard to spotlots changed, nothing written down — and answers vary anyway

The parrot, the note and the shelves keep changing with no record. And parrot answers vary anyway, so a breakdown is hard to spot.One line of the note, one shelf of the library, a new parrot version — change any one and answers shift. And with the silliness dial, the same question already varies a little each time, so eyes alone can't tell a breakdown.

NOTE v4v3 → v4 loggednote versionsCHAT LOG09:00 settle?09:01 answer ✓log the chatsmeasure beans and timedaily carethe keeperTESTtoday: 92run the testsring when oddLLMOps: caring for the parrot every day — write, log, measure, test, and ring the bell

LLMOps is caring for the parrot every day: note versions, chat logs, measuring beans and time, running tests, ringing the bell when something is off.Building is not the end; caring is the start. Write down every change, keep every chat, run the tests daily. That is how you can answer since when, and why.

Notes have versions toov3 → v4, who and when
Log chats, mask secret beansnames and numbers blanked
Test before, compare afterscore drops? roll back
Watch cost, time, refusals dailya jump rings the bell
CONTROL BOARDNOTE VERSIONv3 → v4changed today 10:00roll back ✓BEAN COSTup since 10TEST SCORE92 → 88refusals 3%score droppedALARMringing!one board shows it — the note changed, beans went up, the score went down

One board shows it all. The note changed, beans went up, the score went down — the bell rings.Note v4 went in at 10, and from then the bean cost rose and the test score fell. Now you know since when, and why — one roll back fixes it.

the parrot did not changesame birdhow do refunds work?um… settle?the guests ask new thingsanswers get worse — driftso we keep watchingTESTTO DOrun teststry note v5tests, beans, refusals — dailyring the bell, mend the fencethe caring never endseven with the same bird, a changing world spoils answers — so keep watching

Even if the parrot doesn't change, the guests' questions do, and answers get worse (drift). So we keep watching.The test is daily, not once. Ring the bell when the bean cost jumps or more answers hit the fence. And when logging chats, always mask the secret beans.

In one breath

LLMOps = daily care for a parrot that serves guests. Version the notes, log the chats (masking secret beans), measure beans, time and refusals, test before and after changes, ring the bell when something is off. Even an unchanged bird drifts as the world changes, so keep watching.

Everything involved in operating an LLM app. Version prompts like code, trace requests, responses and tool calls into logs (with PII masking), and monitor cost, latency, refusal rate and quality scores. Catch regressions with an evaluation pipeline before deploying, and use canary releases to expose changes to a slice of traffic first. Because quality drifts as the input distribution shifts even when the model doesn't change, evaluation runs continuously.

When grown-ups say it

LLMOps
Caring for the parrot daily. The real work starts after building: write, log, measure, test, ring the bell.
Prompt versioning
Writing v3, v4 on the notes. Keep who changed what and when; roll back to the last version if things get worse. → the trainer's note
Tracing · logs
Logging the chats. Keep the question, the answer, tool notes and timings. Mask the secret beans. → secret beans
Evaluation pipeline
Automatic tests on every change. Change the note or the bird and the test runs by itself. → the examiner's scorecard
Monitoring
Watching cost, time and quality daily. Put bean cost, answer time, refusal rate and test score on the board. → the bean bill
Drift
Same bird, worse answers. Happens when the guests' questions or the world change. That is why testing is continuous, not once.
Canary release
Just one guest in ten first. Show the new note to a few guests; if the bell stays quiet, roll it out to the rest.
Guardrail
The fence. A sudden rise in answers hitting the fence means something changed. → the fence
Next story
The Lean in the Parrot's Books →
Want another term as a picture book? Request →