Explain like I'm five · AI Security

What is a Model Extraction Attack?

A model extraction attack, told as a story about asking question after question to quietly copy the whole recipe book.

another question again today — but each one looks ordinaryeach single ticket is just a normal order

A guest keeps sending thousands of tickets.Each single ticket looks just like a normal order — nothing seems odd about it.

answers one ticket at a timethe chef has no memory of how many thousand times this same guest has asked

The chef only sees one ticket at a time and never sees the whole pattern.The chef has no memory of how many thousand times this same guest has asked.

sends tens of thousands of ticketsanswers each one, ordinarilythe answers are pieced into a lookalike recipe bookasking question after question to quietly copy the whole recipe book

Asking question after question to quietly copy the whole recipe book.Without ever stealing the book itself, the collected answers build something almost identical.

Flood it with questionstens of thousands of slightly varied asks
Collect every answerfind the pattern in the piled-up answers
Build a lookalike booka copy that tastes almost the same
Copy it for a fraction of the costyears of ingredient cost collapse into weeks
TIMELINE OF A COPYday 1 — send 100 ticketsday 10 — 10,000 tickets piled upday 30 — the answer pattern is fully visibledone — a fake recipe book that tastes almost the samea costly recipe book, copied for next to nothing

The pile of tickets grows into a fake recipe book.A costly recipe book, copied for next to nothing.

just one rule slips outa single rule leakedasking tens of thousands of times clones the whole bookthe whole recipe clonedthe scale differs — one leaked line versus the whole book cloned

A single leaked rule is different from the whole recipe being cloned.The scale differs — one leaked line versus the entire book cloned.

In one breath

Model extraction = cloning the whole recipe book in secret — not by stealing it, but by flooding it with questions and piecing the answers together.

A model extraction attack sends a flood of queries to an API and uses the collected responses to train a substitute model that behaves like the original. Rather than stealing the weights themselves, it clones the model's behavior — the input-output mapping — which makes it hard to detect and effectively steals intellectual property and training cost. Defenses include rate limiting, adding slight variation to responses, detecting unusually diverse query patterns, and watermarking.

When grown-ups say it

Model extraction attack
Cloning the whole recipe book by collecting answers. Not stealing the book — imitating it.
Prompt leak
Just one rule slipping out. Different in scale and method — one leaked line versus the whole book cloned. → fishing out the secret rule page
Query flooding
Sending an endless stream of tickets. The first step in gathering the data needed to clone.
Distillation
Building a small recipe book by studying a big one. With permission it's distillation; done secretly, it's extraction.
Open weights
Publishing the recipe book openly from the start. Not a secret copy — made public from the start. → What are Open Weights?
Rate limiting
Capping how many tickets one guest can send per day. Makes tens of thousands of asks much harder to pull off.
Watermarking
Leaving a small mark in answers to recognize a copied book later. Lets you check later whether a clone is circulating.
Red team
Testing by deliberately flooding it with tickets. AI red teams hunt for this too. → the security world's red team
Next story
A Face Rebuilt from a Pile of Dishes →
Want another term as a picture book? Request →