Explain like I'm five · AI

What is Inference?

Inference, told as a story about what it costs, in beans and time, when a parrot done with school answers a guest one bean at a time.

parrot school — once, big₩our house — beans every questionschool is over, so now it is free, right?school happens once, but every answer costs beans — the guest does not know

Parrot school happens once. But every time a guest asks, beans are spent.The guest thinks it is free once school is over. It is not — every single answer has a price.

huge × oncetraining = readingonce at school, bigTodayissunnysmall × 1,000,000inference = answeringat home every time, small — but millions of timessmall once, big a million times — answering can cost more than school did

Training is reading — once, big. Inference is answering — small, but millions of times.Small once, big a million times. A busy house can end up paying more for answers than the school cost.

₩time · costa parrot done with schoolone bean at a time — each costs time and moneyhow is the weather today?WHAT RAISES COSTmore beansbigger parrotmore mutteringinference: a parrot done with school answers a guest, making one bean at a time

Inference is a parrot done with school answering a guest, making one bean at a time.Cost and time grow with the number of beans, the size of the parrot, and how much it mutters before answering.

Beans out cost more3 to 5 times beans in
Big parrot: pricier per beanand slower too
First bean and last bean differthe wait vs time per bean
Same start, prepared aheadbeans laid out ahead are cheap
questionfirst beanendtime to first beantime per bean × beanstotal timeCOST EXAMPLE1,000 beans in × 1 = 1,000300 beans out × 5 = 1,500 → total 2,500the first bean is the wait, then it is time per bean — cost counts beans

Time splits into the wait for the first bean and time per bean; cost counts beans.Showing the first bean as soon as it comes is streaming. A repeated start gets cheaper with beans laid out ahead.

correct (first try)₩₩₩₩₩5 in beansbig parrot: slow, pricier per bean, but oncetotal 5wrong… again… and again₩₩₩₩₩₩2 in beans × 3 triessmall parrot: fast and cheap, but three triestotal 6 — it cost morefast is not always cheap — pick the parrot that fits the job

Fast is not always cheap. A small parrot that fails several times costs more.The price of beans is the pricing story; which parrot gets the job is routing; making a parrot lighter is quantization.

In one breath

Inference = a parrot done with school answering a guest one bean at a time. School happens once, but every answer costs, and cost and time grow with beans, parrot size, and muttering.

Running a trained model on real inputs to produce outputs. Latency splits into time to first token (TTFT) and time per token; cost scales with input and output token counts and model size. Accelerators such as GPUs batch many requests to raise throughput. Over time, inference often costs more than training did.

When grown-ups say it

Inference
The parrot answering. After school, it makes beans one at a time for a guest.
Training vs inference
Reading vs answering. Reading is once and big; answering is small each time — but millions of times.
Latency
Time for one answer. The wait for the first bean plus time per bean × beans.
Throughput
Answers per hour. Several parrots, or one parrot serving several guests at once.
Time to first token
The wait for the first bean. Show the first bean as soon as it comes. → showing beans one by one
GPU · accelerator
The fast perch the parrot sits on. Makes beans fast. Pricey, and seats are limited.
Batching
Several guests at once. Answering gathered questions together lowers the price per bean, at the cost of a small wait.
Cost
The price of beans. Beans in plus beans out, times the rate. → the price of beans
Next story
The Price of Beans →
Want another term as a picture book? Request →