What is Memorization?
Memorization, told as a story about the chef serving a line straight out of the recipe book — letter for letter — instead of cooking something new.
Asked for a poem, and a line from the recipe book comes out word for word instead.Something completely different was asked for, yet an unrelated sentence pops out exactly as written somewhere else.
The rarer yet more repeated a line is, the easier it is to memorize whole.It isn't summarizing or rephrasing by understanding. A line seen often enough simply gets stuck in its memory, whole.
Memorization is the habit of serving a line straight out of the recipe book instead of cooking something new.It isn't lying on purpose. It genuinely believes that line is the right answer, and serves it exactly as remembered.
If the exact same sentence comes out, that isn't something new — it's copied.Short, rare, often-repeated lines are the riskiest — phone numbers, passwords, and unique paragraphs especially.
The policy asks why to be careful — memorization is how it actually leaks.AI privacy covers the overall risk of handing over a whole list. Memorization is one concrete way that risk actually plays out — and both model inversion and membership inference only work because something got memorized in the first place. A human chef would notice "wait, isn't this copied?" — this one believes the memorized line really is the right answer.
Memorization = the chef's habit of serving a recipe-book line letter for letter instead of understanding and cooking something new. Short, rare, often-repeated lines are the ones most likely to leak out this way.
The phenomenon of an LLM reproducing part of its training data nearly verbatim in its output, instead of summarizing or generalizing it. Memorization grows more likely with larger models, data that repeats the same line often, and lines that are rare. Detection works by feeding in a prefix of a known training line and checking whether it continues it exactly (an extraction-attack test); defenses include deduplication, output-side similarity checks, and excluding sensitive data from training. Both model inversion and membership inference depend on memorization happening in the first place.
When grown-ups say it
- Memorization
- The habit of serving a line straight out of the recipe book. Not cooking something new — just copying it exactly.
- Model Inversion
- An attack that sketches the original ingredients backward from the dishes. The better it memorizes, the easier it is to invert. → a face rebuilt from a pile of dishes
- Membership Inference
- An attack that finds out only whether a guest was part of practice. That eyes-closed, instant reaction is itself a trace of memorization. → the dish made with eyes closed
- AI Privacy
- The policy-level question of what happens when you hand over the whole list. Memorization is how that risk actually happens. → the secret beans we gave the parrot
- Hallucination
- Confidently making things up when it doesn't know. Same root, a different face — instead of making things up, this time it copies exactly. → What is a Hallucination?
- Fine-tuning
- The short drill. The narrower and more repetitive the drill data, the more memorization happens. → the short drill
- Red Team
- The people who probe it with marked test lines first. They check before release whether anything comes out verbatim. → the security world's red team
- Deduplication
- Not putting the same line into the recipe book over and over. Fewer repeats mean less risk of anything getting memorized whole.