What is Data Poisoning?
Data poisoning, told as a story about a suspicious sack slipped into the pantry before the chef ever starts practicing with what's inside.
The supplier delivers sacks of ingredients — and one of them looks off.From the outside it looks just like every other sack. Without opening it, there's no way to tell.
It believes anything that made it into the pantry is fair ingredients to practice with.A human chef would open each sack and look. This one trusts everything in the pantry equally.
Data poisoning is slipping a suspicious sack into the pantry before practice even starts.It isn't tampering with a finished dish — it's tampering with the very ingredients used to practice. That's why the habit gets twisted from the start.
A habit twisted this way gets baked in through all that practice, and is hard to undo.Even a single tainted sack, given enough practice, can settle in and start to feel like an ordinary habit.
A habit already baked in only changes with fresh practice — and it matters that this mixes in at the ingredient stage.Planting a habit later, while it's being refined (fine-tuning), rather than at the ingredient stage, is a different story. That's why the supplier has to be trustworthy from the start.
Data poisoning = slipping a suspicious sack into the pantry before practice (training) even starts. Once a habit is baked in through all that practice, it's hard to undo.
A supply-chain attack that secretly mixes malicious examples into a model's training (or fine-tuning) data, degrading overall quality or making it respond incorrectly on purpose for specific inputs. Crawled web data, external datasets, and user-feedback loops are all possible injection paths. Defenses include verifying data provenance, sampling and auditing, statistical outlier detection, and logging where data came from. Once it's baked into the trained weights, retraining is the only way to undo it.
When grown-ups say it
- Data Poisoning
- A suspicious sack secretly mixed into the pantry. It gets mixed in at the ingredient stage, before training even starts.
- Supply Chain Attack
- The problem that starts when the supplier itself can't be trusted. The general concept in the security world. → What is a Supply Chain Attack?
- SBOM
- A table noting where each sack came from. Tracing it back after the fact needs this table. → the ingredient provenance table
- Fine-tuning
- The short drill. The same mixing-in can happen even in a narrow drill. → the short drill
- Open Weights
- Our own parrot, whose origin we can actually check. A borrowed model makes checking the ingredients' origin harder. → our own parrot vs a rented one
- Red Team
- Deliberately testing with a suspicious sack on purpose. It catches this kind of mixing-in before release. → the security world's red team
- Memorization
- The habit of blurting out what it memorized, word for word. A tainted sack repeated often enough can even get memorized whole. → the dish that's a carbon copy
- Trojan Model
- A habit planted later, while it was being refined (fine-tuning). Planted at the refining stage, not the ingredient stage — that's the difference. → a habit planted while it was being refined