Explain like I'm five · AI

What is Multimodal?

Multimodal models, told as a story about a parrot that turns pictures and sounds into beans too.

I only read words… no photos12,800receipt photothis receipt — what is the total?hold up a photo and the parrot might as well be blind

The guest holds up a receipt photo. A words-only parrot cannot see it.Tables, charts, photos — all the same. The parrot might as well be blind.

a bird that only ate beansreceiptphoto…12,800not a beanCAN EATword beans ✓pictures ×sounds ×the numbers in the photo never become beansthe beak is shaped for beans — a photo will not fit as it is

Because the parrot was a bird that only ate beans. A picture is not a bean.The beak is shaped for beans. The numbers in the photo never become beans, so the parrot does not even know they are there.

12,800photohundreds of beansthe total is 12,800 wonhowmuchis itsounds become beans tooguestEATSwords ✓ pictures ✓sounds ✓multimodal is a parrot that turns pictures and sounds into beans too

Multimodal is a parrot that turns pictures and sounds into beans too.One photo becomes hundreds of beans. Then it can continue them just like word beans — as if it were looking at the photo.

Hand over the photo as isshow it, do not retype it
Pictures are many beansthey fill up the tray
Reads words in picturessigns, tables, handwriting
Drawing is another parrotpulling pictures from fog
cut into patchesa few beans per patchone picture = hundreds of beans → the tray fills fasthellosound → beansOPEN 9-6OPEN9-6words in a picture become beans tooanswering with a picture is another parrot's job

A picture becomes beans patch by patch. So do sounds, and words inside pictures.One picture is hundreds of beans, so the tray fills fast. Answering with a picture is another parrot's job.

the March bar is tallestJanFebMarbig shapes it reads wellthe total is 12,30012,800(really 12,800)small print and digits go wrong easilyit does not see — it continues beans made from the picture; a person checks the numbers

Big shapes it reads well. Small print and digits go wrong easily.It does not see — it continues beans made from the picture, and when the beans are blurry it fills in plausibly. A person checks the receipt numbers.

In one breath

Multimodal = a parrot that turns pictures and sounds into beans too. One photo is hundreds of beans, so the tray fills fast, and a person checks small print and digits.

A model that takes images, audio, or video as input (sometimes output) besides text. Images become image tokens patch by patch and use a lot of the context window; speech becomes tokens through STT. Image generation is usually a separate diffusion model.

When grown-ups say it

Multimodal
The parrot that sees pictures. Words, pictures, and sounds all become beans for one parrot.
Vision model
The picture beak. The part that turns pictures into beans. It sits attached to the big parrot.
Image tokens
Picture beans. One photo is hundreds to thousands of beans, so the tray fills fast. → the tray
Speech-to-text · text-to-speech
Sound → beans, beans → sound. STT turns speech into beans; TTS turns beans back into speech.
Difference from OCR
Pulling out letters vs eating the whole picture. OCR peels off only the letters. Multimodal reads letters, shapes, and layout together.
Document understanding
Reading tables, receipts, contracts. It reads where each cell sits. Still, a person checks the numbers.
Video input
Many pictures in a row. A video becomes many picture frames turned into beans — a lot of beans.
Image generation
The drawing parrot. Seeing and drawing are different jobs. → pulling pictures from fog
Next story
Pulling a Picture Out of the Fog →
Want another term as a picture book? Request →