Explain like I'm five · AI

What is a Token?

Tokens, told as a story about a parrot that reads and counts words not in letters or words, but in beans.

why does a long text cost more?not letters — beansTHE BILLshort: 5 beanslong: 300 beansthe parrot counts words in beans — more beans, more money and time

The guest asks: why does a long text cost more?The parrot doesn't count letters. It counts beans. More beans means more money and a longer wait.

안녕하세요안녕하세요4 beanshellohello1 beanunbelievableunbelievable3 beanspieces, not wordsa short word can be many beans, a long one just one — Korean usually cuts finera bean is neither a word nor a letter — it is the parrot's own piece

Beans split differently from words.A short greeting is four beans, a long English word is three. Korean usually splits finer, so the same meaning takes more beans.

Today the weather is niceTodaytheweatherisnice5 beansBEAN PRICE1000 beans= one coinpriced by the beana token is the bean the parrot counts words in

A token is the bean the parrot counts words in.Words coming in are split into beans; answers come out one bean at a time. The bean count is the price and the time. Only as many as fit on the tray — see the tray in front of the parrot.

Count in beansnot letters, not words
Korean cuts finersame meaning, more beans
Beans = cost and timebeans in and beans out
Only what fits the traybeans that fit at once
same words: Today the weather is nice5 beans7 beans9 beansWHY DIFFERENT?each machine cutsin different placesKorean: usually finerdifferent bean counters — different counts, different pricesevery parrot has its own bean counter

Every parrot has its own bean counter.Same sentence, yet one parrot counts five beans and another nine — they cut in different places. So the price differs too.

bankriverbank= a shoremoneybank= a shopa bean alone has no meaning — its neighbours decideit only sees the traythe tray = beans that fit at oncethe tray has its own pagea bean is for counting — meaning comes from continuing, the limit from the tray

One bean isn't one meaning. And the tray has a limit.The bean bank alone doesn't say shore or shop; continuing from the beans around it decides. How many beans fit at once is decided by the tray.

In one breath

Token = the bean the parrot counts words in. It's a piece, neither letter nor word, and Korean splits finer. The bean count is the price and the time.

The smallest unit an LLM reads and writes. A tokenizer splits text into tokens by subword rules (BPE and the like); rules differ per model, so the same sentence has different token counts. Input and output token counts set price and latency, and the context window is capped in tokens.

When grown-ups say it

Token
One bean. The piece the parrot counts words in. Often smaller than a word.
Tokenizer
The bean counter. The rule that cuts a sentence into beans. Different for every parrot.
Subword (BPE)
A word piece. Letters that often go together become one bean. BPE is the usual method.
Token count and cost
The bean price. Beans in and beans out both count. More beans cost more and take longer.
Context window
The tray. How many beans fit at once. → the tray in front of the parrot
Korean token efficiency
Korean cuts finer. The same meaning takes more beans than English — and costs more.
Embedding
A coordinate tag on each bean. The parrot turns each bean into a bundle of numbers; similar meanings get nearby tags.
Prompt
The trainer's note. The note is counted in beans too. → the trainer's note
Next story
The Note the Trainer Hands the Parrot →
Want another term as a picture book? Request →