Alerting Without Alert Fatigue

When alerts fire constantly, people stop reading them, and the one that matters gets missed. Good alerting is mostly about deciding what not to page on.

1. Every page must be urgent and actionable

Ask: "If this fires at 3 a.m., does someone need to do something right now?" If not, it's a ticket, a dashboard or a daily report, not a page.

2. Alert on symptoms, not causes

• Page on: error rate above SLO, checkout latency too high, queue age growing, no successful jobs in an hour.
• Don't page on: CPU at 85%, one pod restarted, a disk at 70%. Show those on dashboards to explain a symptom.

3. Use durations and sensible thresholds

# Prometheus alerting rule
groups:
  - name: checkout
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum(rate(http_request_duration_seconds_count{route="/checkout",status=~"5.."}[5m]))
            / sum(rate(http_request_duration_seconds_count{route="/checkout"}[5m])) > 0.02
        for: 10m                   # must stay true for 10 minutes
        labels:
          severity: page
        annotations:
          summary: "Checkout error rate above 2% for 10 minutes"
          runbook: "https://wiki.example.com/runbooks/checkout-errors"

4. Write a runbook for every alert

• What the alert means and the user impact.
• The dashboards and queries to check first.
• Known causes and how to fix or mitigate them (roll back, scale up, fail over).
• Who to escalate to.

5. Route and group

• Send alerts to the team that owns the service.
• Group related alerts so one incident creates one notification, not fifty.
• Silence alerts during planned maintenance.

6. Review regularly

Once a month, look at every alert that fired. Delete alerts nobody acted on, fix flaky ones, and add alerts for incidents that users noticed before you did.