Explain like I'm five · AI

What is Attention?

Attention (and the transformer), told as a story about a parrot that glances at every bean on the tray and looks longer at the related ones before picking the next.

the bank sat by the river — which bank?the river bank!not the money bankit used the far-away river to get it right — how?

A confusing word — and the parrot got it right without getting confused.Bank could mean money or river. The parrot chose by looking at river, far back in the sentence. How?

the old parrot only looked at the last few beansDownriverthebankstoodwhichsees only theseforgotten?the money bank!it forgot the far river bean and chose from the near bank beanread only in order, and far-away beans vanish

The old parrot looked only at the last few beans. Far-away beans were forgotten.Read the beans one by one in order, and the first ones fade by the end. So it forgot river and picked the money bank.

it glances over every bean on the trayrelated beans get a thicker gazeDownriverthebankstoodwhichbank?inside its headgaze layer 3gaze layer 2gaze layer 1stacked up = a transformerthick line = lots of gaze, thin = a littleattention: when picking the next bean, glance at every bean and look longer at the related ones

Attention: when picking the next bean, the parrot glances at every bean and looks longer at the related ones.The thickness of the lines between beans on the tray is the gaze. A head that stacks this gaze in layers is a transformer — every modern parrot is built this way.

Sees every bean at onceall at once, not in order
Thicker gaze for related beansfar but related still thick
Many layers of gazeeach layer looks differently
More beans, more gazethis is where tray cost comes from
layer 1: neighbours12345678← order tagslayer 2: meaning pairslayer 3: whole sentencefar apart, still linkedeach layer looks differently — and order tags tell it which bean came first

Each layer looks differently. Order tags on the beans tell it which came first.If it sees all beans at once, how does it know the order? Each bean wears a number tag. And within one layer there are several gaze strands (heads), each watching a different kind of relation.

next to river, so this bank!the gaze measures which beans relateit still does not know the meaning4 beans → 6 lines8 beans → 28 linesdouble the beans, four times the gaze — why long trays costmeasuring relatedness is not understanding — and gaze is not free

Measuring relatedness is not understanding. And gaze is not free.Double the beans and the lines between them quadruple. That is why a long tray is slow and costly — and why saving beans matters. That the parrot still does not know meaning is in the first story.

In one breath

Attention = when picking the next bean, glance at every bean on the tray and look longer at the related ones. A head that stacks this gaze in layers is a transformer. More beans means gaze cost grows with the square.

An operation where each token in a sequence computes a relatedness score (query-key dot product) with every other token and forms a weighted sum. Within one sequence it is self-attention; split into several strands it is multi-head. Position is added separately via positional encoding. Stacking these layers gives the transformer, and because the cost scales with the square of sequence length, long contexts are expensive.

When grown-ups say it

Attention
The parrot's gaze. Glance at every bean, look longer at the related ones, then pick the next.
Transformer
A head of stacked gaze layers. Nearly every modern parrot has this head. The T in GPT is this.
Self-attention
Beans on the tray watching each other. The gaze goes between beans in the same sentence, not anywhere else.
Layer · Head
Gaze layers and gaze strands. Layers stack upward; heads are strands within one layer. Each watches a different kind of relation.
Positional encoding
The number tag on each bean. Seeing everything at once loses the order, so each bean gets a tag.
Quadratic cost
Double beans, four times the gaze. Because every bean connects to every other. Why long trays cost.
Context window
The tray. The gaze only reaches beans on the tray. → the tray in front of the parrot
LLM
The mountain-of-books parrot. The bird that read its books with this gazing head. → the parrot that read a mountain of books
Next story
The School of Reading →
Want another term as a picture book? Request →