Alerting & Runbooks
Alert thresholds, alert fatigue, P1/P2/P3 classification, runbook structure
The monitoring system fires 47 alerts in a week. The team acknowledges every one and fixes none. The alerts have become wallpaper — everyone has learned to tune them out. Two months later a real model failure runs undetected for three days, because the alert *did* fire and nobody acted. Alert fatigue has quietly converted the whole monitoring system into a false sense of safety.
Fatigue is a calibration problem, not a people problem
When the false-positive rate is high, ignoring alerts is the *rational* response — engineers are running expected-value math, not being lazy. An alert that's actionable only 20% of the time trains everyone to assume it's noise 4 times out of 5. The first genuine P1 that gets waved off during that mental shortcut is where fatigue turns into real business damage.
The fix starts with runbooks
A runbook is the procedure stapled to an alert type: what triggered it (the exact condition), what it means (the business interpretation), its severity, the immediate action (page? auto-rollback?), the ordered investigation steps, and the resolution decision tree (if X then Y, else Z). Without one, every alert forces the on-call engineer to re-derive the whole investigation from scratch at 3am. A runbook turns one person's expertise into a process anyone on the rotation can run.
Four rules that keep false positives down
*Actionable* — if there's no clear action when it fires, it's a metric to watch, not an alert to page on. *Low false-positive rate* — past ~20%, people start ignoring it. *Severity routing* — P0 (model down / financial risk: page now), P1 (significant drift: ticket for tomorrow), P2 (early warning: weekly queue). *Deduplication* — collapse 15 cascading alerts from one upstream failure into a single alert with a root-cause hypothesis.
And the myth to bury: "more alerts = better monitoring." More alerts means more noise means ignored alerts means *worse* monitoring than having fewer. The target is zero false positives, every alert actionable, every alert backed by a runbook. Start with five high-signal alerts and add another only when it has a written runbook and a measured false-positive rate under 20%. Never enable an alert you haven't written the runbook for.
Key points
- Write the runbook before enabling the alert — if you cannot write the runbook, you do not understand the alert well enough to act on it. A runbook takes 30 minutes to write and saves hours per incident. The writing process itself forces you to answer the question: if this fires at 3am, what exactly does the on-call engineer do? If you cannot answer that, the alert is not ready to ship.
- Trap: setting uniform alert thresholds across all models and features. A PSI threshold of 0.2 appropriate for a stable user behavior feature will fire constantly for a feature that naturally varies with seasonality. Calibrate thresholds per feature based on observed historical variation from the first 30 days of production traffic. A threshold calibrated to the feature's natural variation generates one-tenth the false positive rate of a uniform threshold.
- Diagnostic: track alert-to-action conversion rate monthly. If more than 40% of alerts result in "no action taken," those alerts are generating noise. Tighten thresholds, add context to help engineers triage faster, add duration requirements (must persist for 1 hour before paging), or deprecate the alert entirely. An alert with zero true positives in 3 months is not a safety net — it is alert debt that degrades the team's response to real incidents.
Alert fatigue is a calibration problem, not a personnel problem — write the runbook before enabling the alert, calibrate thresholds per feature from observed production variation, and track alert-to-action conversion rate monthly to catch alert debt before it degrades incident response.
Recap
- Alert fatigue is calibration, not people: ignoring noisy alerts is rational expected-value math.
- High false-positive rate trains dismissal: 20%-actionable alerts get waved off 4 in 5 times.
- Write the runbook before enabling the alert — if you can't, you don't understand it well enough to act.
- Runbook = trigger + meaning + severity + immediate action + investigation steps + resolution tree.
- Four rules: actionable, false-positive rate <20%, severity routing (P0 page / P1 ticket / P2 weekly), deduplication.
- Calibrate thresholds per feature from 30 days of production variation — uniform PSI 0.2 spams seasonal features.
- Track alert-to-action conversion: >40% "no action" is noise; start with 5 high-signal alerts, target zero false positives.
Check your understanding
Q1. Your on-call engineer receives 50 alerts per day, most of which turn out to be false positives. How do you fix this?
- A) Grow the on-call rotation so each engineer handles fewer alerts — the volume is fine if it's spread out
- B) Disable every non-P1 alert immediately and rebuild the whole system from scratch with stricter gates
- C) Audit 30 days of alerts, tighten thresholds, drop zero-true-positive alerts, target under 5/day
- D) Make every alert non-paging and require morning dashboard checks — this loses no real signal coverage
Q2. Which two statements correctly explain why the module insists you write the runbook before you enable the alert?
- A) If you can't write what the on-call engineer does when it fires, you don't understand the alert yet
- B) The writing process itself forces you to answer the concrete 3am question before the alert can page
- C) The runbook is a compliance artifact auditors require on file before any alert can legally page anyone
- D) Runbooks take longer to write than alerts take to configure, which balances the sprint workload
Q3. A single upstream feature-store outage triggers 15 cascading alerts at once. Which of the four rules addresses this directly?
- A) Actionability — none of the 15 has a clear action, so downgrade them all to dashboard metrics instead
- B) Severity routing — route all 15 to the P0 lane so on-call sees the outage and begins triage at once
- C) Deduplication — collapse the 15 cascading alerts into one carrying the root-cause hypothesis instead
- D) False-positive rule — cascading alerts are false positives by definition, so raise every threshold up
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →