Explain like I'm five · AI

What is Quantization?

Quantization, told as a story about rewriting the fine numbers in a parrot's head on a coarse scale so it gets light.

the big parrot — heavywant to put it here…×our small perch (phone, laptop)the big parrot will not fit on the small perch — no room

We want the big parrot on our small perch (phone, laptop), but it is far too heavy.It can only live on the big perch at parrot school. To bring it home, it has to get lighter.

inside its headNUMBERS IN ITS HEAD0.48231937-0.139204710.77219003… ×billionseach one to eight decimal places0.48231937 kgan eight-decimal scalethe finer, the bigger and heavierbillions of numbers, each written finely — that takes a lot of room

The parrot's head holds billions of numbers, each written very finely.An eight-decimal scale is big and heavy. Every number is written on a scale like that — so the parrot is heavy.

fine tickscoarse ticksNUMBERS0.4823-0.1392finely written — heavyrewrite coarselyNUMBERS0.5-0.1coarsely written — light, a bit dullerquantization: rewrite the fine numbers on a coarse scale, and the parrot gets light

Quantization rewrites the fine numbers on a coarse scale, and the parrot gets light.Like writing 0.4823 as 0.5. It gets a little duller, but far smaller and faster. It is the same parrot, not a new one.

Coarser ticks: lighter, duller8-bit a little, 4-bit more
Test it on our jobhow much did the score drop
Nearly a must on phones, laptopssmall perch, light bird only
Do it after the drillteach first, then coarsen
0.48231937 kgfine scale · heavy0.5 kgcoarse scale · lightTEST SCORE (OUR JOB)original · 100 · 16 GB8-bit coarse · 99 · 8 GB4-bit coarser · 96 · 4 GB2-bit very coarse · 70 · 2 GBcoarser ticks: lighter, and the score slips a little8-bit and 4-bit are usually fine; too coarse and it gets dull — pick by test

The coarser the ticks, the lighter the parrot — and the score slips a little.8-bit and 4-bit are usually fine; too coarse and it goes dull fast. Pick how coarse by testing. Do the drill (fine-tuning) before you coarsen.

summary: the meeting is at 3FINE JOBSsummary · translation · sortingshort answersnearly the same even when coarse12 × 13 = 146… maybe?SENSITIVE JOBSmath · long reasoninglong codesome jobs go dull fast on coarse ticksif coarse ticks fall short, take the other road: teach a new small parrot (distillation)

Summaries and translation survive coarse ticks; math and long reasoning can go dull fast.If coarse ticks fall short, there is another road — distillation, teaching a new small parrot. What the lighter parrot costs per answer is the inference story.

In one breath

Quantization = rewrite the fine numbers in the parrot's head on a coarse scale, making it far smaller and faster. It gets a little duller, so test on our job to pick how coarse.

Converting model weights from high-precision formats like FP16 or BF16 to lower bit widths such as INT8 or INT4 to cut memory and inference cost. There is a precision loss, so verify with benchmarks and apply it after fine-tuning. Nearly mandatory for on-device (phone, laptop) deployment.

When grown-ups say it

Quantization
Rewriting numbers on a coarse scale. The same parrot gets light and fast. A little duller.
Bits (FP16 · INT8 · INT4)
How coarse the ticks are. 16-bit is fine, 8-bit coarse, 4-bit coarser. Smaller number: lighter and duller.
Precision loss
How much duller it got. Writing 0.4823 as 0.5 is slightly off. It shows most in math and long reasoning.
On-device
Our small perch at home. Running the parrot right on a phone or laptop. Only a light parrot fits.
Inference speed
Time taken to answer. A lighter parrot answers faster. → what an answer costs
Memory
The room the parrot takes. A 16 GB parrot becomes 8 GB at 8-bit, 4 GB at 4-bit.
Versus distillation
Lighten the same bird vs teach a new one. Quantization coarsens the numbers; distillation raises a new small parrot. → teaching the small parrot to copy
Evaluation
The test after coarsening. Check on our job how much the score dropped. → the examiner's scorecard
Next story
What an Answer Costs →
Want another term as a picture book? Request →