Cloud Parts

Alerting

Rules that watch your metrics and logs and wake a human only when users are, or are about to be, hurt.

4 technologies on this page ↓
Live model
What you’re watchingServices send metric samples to Prometheus, which evaluates two rules. Most evaluations pass quietly; now and then one fires, and Alertmanager routes it by severity: the user-facing error rate pages the on-call phone, the filling disk goes to the team channel.

How it works

An alerting rule is a query plus a condition, evaluated on a schedule, typically every minute: “the share of checkout requests that fail is above 5%”. To ignore blips, the condition must hold for a while before the alert fires: a for: 10m clause in Prometheus, or “3 out of 5 data points” in a CloudWatch alarm. Until then the alert is pending. A firing alert carries labels such as service and severity, and a notification layer decides what happens next. It groups related alerts into one message (fifty instances losing their database become one page, not fifty), routes by label (severity=page to the on-call phone, severity=ticket to a team channel), inhibits alerts that follow from a bigger one (no per-service pages while the whole cluster is unreachable) and honours silences during planned work. What to alert on matters more than how. Alert on symptoms users feel, such as errors, latency and unavailability, rather than causes like high CPU, which may be harmless. A stronger form alerts on the error budget of a service level objective (SLO): with a 99.9% target, failing at 14.4 times the allowed rate for an hour burns 2% of a 30-day budget and pages someone, while a slow burn opens a ticket. Every page should be urgent and actionable; alerts people learn to ignore are worse than none.

Think of it as

A smoke detector, not a thermometer on every wall. It stays quiet through toast and steam, sounds only when there is a real fire, and the whole building hears one alarm rather than a beep from every room.

Reach for it when

  • Anything users depend on outside office hours
  • Catching symptoms: error rate, latency, failed checks, an SLO burning too fast
  • Slow-building problems (a disk filling, a certificate expiring) routed to tickets, not pages

Think twice when

  • Paging on causes such as CPU, memory or one failed pod when users aren’t affected
  • Alerts nobody can act on, or that are routinely acknowledged and ignored
  • Thresholds copied from a template without looking at the service’s normal behaviour

A real example

A checkout service has a 99.9% availability SLO over 30 days. A bad deploy makes 10% of requests fail, 100 times the allowed rate. The burn-rate alert needs both the 1-hour and the 5-minute error ratios above 1.44% (14.4 × 0.1%); the 1-hour average crosses that about nine minutes in, and the on-call engineer gets one page. A separate rule saw a database disk at 91% that morning; it opened a ticket in the team channel and paged nobody.

Used for alerting

Alerting is something products do, not a kind of product. These are often used for it, grouped by what they are.

What each provider calls it

AWS
CloudWatch alarms (metric, log and composite alarms) with SNS notifications
Google Cloud
Cloud Monitoring alerting policies (metric, log and PromQL conditions) with notification channels
Azure
Azure Monitor alerts (metric, log search and activity log alert rules) with action groups
Open source & others
Prometheus alerting rules with Alertmanager, Grafana Alerting; hosted: Datadog monitors, Better Stack, Sentry alerts

Grouping, routing and inhibition in Alertmanager#

Rules decide when an alert fires; Alertmanager decides who hears about it and how often. This config sends pages to PagerDuty, everything else to Slack, and mutes per-service alerts while their whole cluster is unreachable.

yaml
route:
  receiver: team-chat
  group_by: [alertname, cluster]   # one message per alert name and cluster
  group_wait: 30s                  # wait for related alerts before the first message
  group_interval: 5m
  repeat_interval: 4h              # remind while it is still firing
  routes:
    - receiver: on-call-pager
      matchers:
        - severity="page"

inhibit_rules:
  - source_matchers: [alertname="ClusterUnreachable"]
    target_matchers: [severity=~"page|ticket"]
    equal: [cluster]

receivers:
  - name: on-call-pager
    pagerduty_configs:
      - routing_key: <secret>
  - name: team-chat
    slack_configs:
      - channel: "#checkout-alerts"

In code · yaml

# Prometheus alerting rules (rules.yml)
groups:
  - name: checkout
    rules:
      # symptom, sustained: users see errors for 10 minutes
      - alert: CheckoutHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout", code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="checkout"}[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: More than 5% of checkout requests are failing
          runbook_url: https://wiki.example.com/runbooks/checkout-errors

      # SLO burn rate: 99.9% target, 2% of a 30-day budget gone in an hour
      - alert: CheckoutErrorBudgetBurn
        expr: |
          job:slo_errors_per_request:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
        labels:
          severity: page

Often confused with

MetricsOn-callIncident managementSynthetic monitoring
Where it came from1999–2018

Early monitoring tools such as Nagios ran a check per host or service and sent a notification whenever one failed. As systems grew to hundreds of instances, one network problem could mean hundreds of messages. When SoundCloud described Prometheus publicly in 2015, alerting was split in two: rules written in the query language decided when something was wrong, and a separate Alertmanager handled notifications, aggregated them and let people silence alerts by any set of labels before paging through a service like PagerDuty.

What deserves a page was argued out at Google. Rob Ewaschuk, after seven years on call there as a Site Reliability Engineer, wrote “My Philosophy on Alerting”, widely shared in 2014: pages should be “urgent, important, actionable, and real”, symptoms beat causes, and he could react with urgency only “a few times a day before I get fatigued”. The ideas became the “Monitoring Distributed Systems” chapter of Google’s 2016 Site Reliability Engineering book, which he wrote.

The 2018 Site Reliability Workbook tied alerts to service level objectives. Its “Alerting on SLOs” chapter walks through simple thresholds and their flaws before recommending burn-rate alerts over several windows: page when 2% of a 30-day error budget is spent in an hour (a burn rate of 14.4) or 5% in six hours, and open a ticket when 10% goes in three days. Cloud monitoring services and Grafana now offer the same building blocks: rules, pending periods, routing and silences.

Why it’s built this way

Alert on symptoms, not causes
Users feel errors and latency, not CPU. A symptom rule catches many different causes, including ones nobody predicted, and doesn’t fire when a busy CPU is harmless.
Wait before firing
A for duration or an M-out-of-N rule ignores brief spikes that fix themselves, trading a few minutes of detection time for far fewer false pages.
Separate evaluation from notification
Rules only say what is wrong. Grouping, routing, inhibition and silences in a separate layer turn a storm of related alerts into one message to the right person.
Spend an error budget instead of a fixed threshold
A burn-rate alert asks how fast the service is using up its allowed failures, so a short sharp outage and a long slow leak both get caught, at the urgency each deserves.

Timeline

  1. 1999

    NetSaint, later Nagios, sends a notification per failed check

  2. 2014

    Rob Ewaschuk’s “My Philosophy on Alerting” circulates widely

  3. 2015

    SoundCloud describes Prometheus and its separate Alertmanager

  4. 2016

    Google’s SRE book argues for symptom-based alerting

  5. 2018

    The SRE Workbook recommends multiwindow burn-rate alerts on SLOs

Sources13

Improve this page on GitHub