Explain like I'm five · AI

What is prompt caching?

Prompt caching, told as a story about laying out the usual front beans ahead of time and adding only the new ones.

RULEBOOK (30 pages)1. refunds: 7 days2. delivery: 3 days3. …(same every time)morning, noon, night — every timetrayruleruleruleruleQevery time: bean cost + waitingfrom the top again…the same rulebook goes on the tray every time — paying and waiting every time

The same rulebook goes on the tray every time — and costs money and time every time.The question is one line, but the rulebook is 30 pages. The parrot reads all 30 before answering — and the bean cost comes every time.

the parrot re-reads every bean on the tray from the startruleruleruleruleruleruleQyou pay and wait for every bean it readsthe same thing, again and againBILL100 beans × 3even when the front is identical, the parrot does not know — it reads again

The parrot re-reads every bean on the tray from the start, every time.Even if the front is identical to yesterday, the parrot cannot tell. Every bean on the tray is read fresh, and you pay for each.

beans laid out ahead (already read)only newruleruleruleruleruleQ?skips the front, reads only the new beansI remember the front!traineronly works while the front is identicalprompt caching: lay out the usual front beans ahead, then add only the new ones

Prompt caching lays out the usual front beans ahead of time, then adds only the new ones.The parrot briefly remembers a front part it has already read. For the next guest it skips that part and reads only the new beans — cheaper and faster.

Fixed things go firstrulebook, owner note, samples
Changing things go lastthe guest question at the end
Reorder it and it breaksthe front must match to the letter
It lives only a whileafter some minutes it reads again
NO CACHEruleruleruleruleruleruleQ?COSTall 100 beansslowWITH CACHEruleruleruleruleruleruleQ?COSTonly 20 new beansfastgrey beans = laid out ahead, tiny costsame front, so from the second time it is cheap and fast

With the same front, it is cheap and fast from the second time on.Grey beans cost only a little. The first time costs slightly more to lay them out; from the second time on you win.

FRONT: fixedowner noterulebooksamplesBACK: changesguest questionthe datefixed at the front, changing at the back✓ the cache stays aliveFRONTthe date: 9/14owner noterulebookone character differsfrom the top again!× the cache breaksso the order of notes matters — keep the front identical

Change one character at the front and it starts over. That is why the order of notes matters.Put the date or the guest name at the top of the owner note and it breaks every time. Move what changes to the back.

In one breath

Prompt caching = lay out the usual front beans ahead, then add only the new ones. Fixed things go first, changing things last — if the front differs by one character, it starts over.

Stores the processed result of a prompt prefix for a short time (TTL) and reuses it for later requests that start with the same prefix. On a cache hit, input-token cost drops sharply and latency shrinks. The key is layout: system prompt, long documents and examples first, the user question last.

When grown-ups say it

Prompt caching
Beans laid out ahead. Read the front once, skip it next time. Cheaper and faster.
Cache hit
The front matched exactly. Handled as grey beans. If it does not match, a cache miss — everything again.
Prefix
The front of the tray. How far from the first bean things are identical. Only that part is cached.
TTL (time to live)
It lives only a while. Usually a few minutes. After that it must be laid out again.
System prompt placement
The owner note goes first. The fixed note must sit at the front for the cache to survive. → the trainer note
Context
The tray. The cache remembers the front of the beans on the tray. → the tray
Cost
The bean price. Grey beans are much cheaper. → the bean price
Latency
The wait until an answer. Skipping the front makes the first answer faster. → what answering costs
Next story
Writing in the Boxes →
Want another term as a picture book? Request →