Explain like I'm five · SRE

What is Alerting?

Alerting, told as a story about a pager that only rings when things stay bad enough for guests to feel it.

ringing again... I'll just turn it offthe real emergencynobody noticedthe pager keeps ringing over small stuff, so the on-call turns it off — and misses the real emergency

The pager keeps ringing over small stuff, so the on-call turns it off.And when the real emergency hits, nobody hears it.

they all ring the same — I can't tell what's urgentring for everything, and the important gets lost in the unimportant — that's alert fatigue

Ring for everything, and the important gets lost in the unimportant.That's alert fatigue — eventually everything gets ignored.

the line guests feelit only rings when things stay bad enough for guests to feel ita brief wobble passes — staying bad is what triggers the call

It only rings when things stay bad enough for guests to feel it.A brief wobble passes — staying bad is what triggers the call.

Only what guests would feelbelow that, just let it pass
Let a brief wobble passthat's the duration condition
Grade the rings toojust watch vs call right now
Review every ring afterwardcheck whether it was real
spikes briefly, then comes right back-> doesn't ringstays bad for minutes-> it ringsa duration condition: it has to stay bad for minutes to count

A brief spike (no ring) versus staying bad (it rings).Only staying bad for minutes counts as a real signal.

set the threshold too looseit didn't ring this time...missed this one tooset it too quiet and you miss real problems too — finding the right threshold is ongoing work

Set it too quiet, and you miss real problems too.The right threshold isn't set once — it keeps getting tuned.

In one breath

Alerting = deciding to ring only when things are bad enough for guests to feel — and stay that way — so the pager is saved for real emergencies.

A mechanism that notifies a person when a monitored metric crosses a threshold. Combining a threshold with a duration condition filters out brief wobbles, and severity levels separate 'just watch' from 'page now'. Too many alerts cause alert fatigue, which is its own danger.

When grown-ups say it

Alerting
A signal that rings only for real trouble. Quiet the rest of the time.
Alert fatigue
When everything rings, nothing stands out. Eventually, everything gets turned off.
Threshold
The line that decides when it rings. Too low rings too often; too high misses things.
Duration condition
The rule that ignores a brief wobble. Only staying bad triggers a ring.
Severity levels
Just watch versus call right now. Not every alert carries the same weight.
Burn-rate alerting
An alert that watches how fast the allowed failures are being used up. → the bundle of allowed failure tickets
SLI
The number shown on the gauge itself. It's the value the alert watches. → today's operating record
On-call
Whoever answers when it rings. Whoever is holding the pager this week. → whoever's holding the pager
Next story
The Fake Breakdowns We Cause on Purpose →
Want another term as a picture book? Request →