Cloud Parts

Metrics

Numbers measured over time, such as requests per second, error rate or p95 latency: cheap enough to keep for every service and to alert on.

7 technologies on this page ↓
Live model
What you’re watchingPrometheus pulls. On every interval it scrapes each target’s /metrics endpoint, and the current numbers come back as the reply. A nightly job ends before anyone could scrape it, so it pushes its result to a Pushgateway, which Prometheus scrapes like any other target.

How it works

A metric is a named number sampled over time, with labels that say what it describes: http_requests_total{route="/orders",status="500"}. Three shapes cover most needs. A counter only goes up (requests served, bytes sent) and you chart its rate. A gauge goes up and down (queue depth, memory in use). A histogram counts observations into buckets (requests under 100 ms, under 500 ms and so on), so you can estimate percentiles later and add up the buckets from many servers. The program keeps only running totals and reports them every few seconds, so a metric costs about the same at 10 requests a second as at 10,000. That’s why metrics, not logs, drive dashboards and alerts. The catch is cardinality: every unique combination of label values is a separate time series, so a label such as user_id turns one metric into millions of series and the bill, or the database, falls over. Metrics travel two ways. Prometheus pulls: it scrapes each target’s /metrics HTTP endpoint on an interval (one minute by default, 15 seconds is common), which also tells it when a target is down. StatsD and OpenTelemetry’s OTLP push: the program sends numbers to a collector, which suits short-lived jobs. For what to measure, Google’s SRE book names four golden signals for a user-facing system (latency, traffic, errors, saturation); Tom Wilkie’s RED method (rate, errors, duration) applies the same idea to each microservice, and Brendan Gregg’s USE method (utilization, saturation, errors) to each resource, such as a CPU, disk or connection pool.

Think of it as

A car’s dashboard. The speedometer and fuel gauge don’t record every turn of the wheel, just a few numbers, all the time, so you notice a problem the moment it starts. To find out why, you need the trip log.

Reach for it when

  • Dashboards and alerts: error rate, latency percentiles, queue depth, saturation
  • Trends and capacity planning over weeks and months
  • Service level objectives, which are ratios of good events to all events
  • Very high traffic, where recording every event would cost too much

Think twice when

  • Labels with unbounded values, such as user IDs, emails or raw URLs; each value is a new series
  • Explaining one particular request; that’s what logs and traces are for
  • Averaging percentiles across servers or time windows; add up histogram buckets instead

A real example

An API runs on 20 pods. Each exposes one histogram, http_request_duration_seconds, labelled by route (15 values) and status class (5), with 12 buckets plus a sum and a count: 15 × 5 × 14 = 1,050 series per pod, 21,000 in all, and the number stays the same at ten times the traffic. The p95 alert adds up the buckets from all pods before computing the percentile, because the average of 20 separate p95s isn’t the p95 of the service. When someone proposes a customer_id label for 50,000 customers, the arithmetic says no: about a billion series.

Technologies

Specific products, and what each does differently.

Also used for metrics

Products of other kinds that are often used this way, grouped by what they are.

Observability →

Amazon CloudWatchAWSAWS’s built-in monitoring service: metrics from every AWS service, logs, alarms and dashboards, and now traces, OpenTelemetry and PromQL, billed per metric, per GB and per alarm.Better StackCommercialUptime checks, on-call paging and status pages, plus logs, traces, metrics and errors in a ClickHouse-based store, sold as one SaaS billed per responder and per GB.DatadogCommercialA hosted monitoring platform: an agent on every host ships metrics, logs and traces to Datadog’s SaaS, which bills per host, per GB and per metric.GrafanaOpen sourceDashboards and alerts over data that stays where it is: Prometheus, SQL databases, cloud monitoring and more.OpenTelemetryOpen sourceThe vendor-neutral standard for producing and shipping traces, metrics and logs, with SDKs for most languages and a Collector to route the data.

What each provider calls it

AWS
CloudWatch metrics (Classic, or OpenTelemetry metrics queried with PromQL); Amazon Managed Service for Prometheus
Google Cloud
Cloud Monitoring; Google Cloud Managed Service for Prometheus
Azure
Azure Monitor Metrics; Azure Monitor managed service for Prometheus
Open source & others
Prometheus, StatsD, OpenTelemetry SDKs and Collector, Telegraf with InfluxDB; dashboards: Grafana; hosted: Datadog

What to measure#

Three checklists that overlap on purpose. Use the first two for services and the third for the resources under them.

Four golden signals

Google SRE book

Latency, traffic, errors and saturation for each user-facing system. Measure the latency of failed requests separately: a fast error and a slow success shouldn’t be averaged together.

RED

Tom Wilkie, 2015

Rate, errors and duration of the requests each service handles. Measuring the same three things on every service lets people go on call for code they didn’t write.

USE

Brendan Gregg, 2012

Utilization, saturation and errors for every resource: CPUs, disks, network links, connection pools. Saturation is the extra work a resource can’t service yet, often waiting in a queue.

In code · text

# what a service exposes at /metrics (Prometheus text format)
# TYPE http_requests_total counter
http_requests_total{route="/orders",status="200"} 10432
http_requests_total{route="/orders",status="500"} 17
# TYPE queue_depth gauge
queue_depth 42
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/orders",le="0.1"} 9100
http_request_duration_seconds_bucket{route="/orders",le="0.5"} 10300
http_request_duration_seconds_bucket{route="/orders",le="+Inf"} 10449
http_request_duration_seconds_sum{route="/orders"} 812.4
http_request_duration_seconds_count{route="/orders"} 10449

# PromQL: share of requests failing, and p95 latency, over 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Often confused with

Time-series databasesLogsAlertingObservabilityNarrow tablesLogs, metrics or traces?
Where it came from2003–2026

Watching numbers over time became central at Google first. Shortly after the Borg scheduler was created in 2003, Google built Borgmon to monitor it, and over the next decade moved from check scripts that alerted on failures to collecting time series and turning them into charts and alerts with a rule language. Outside Google, Graphite, written by Chris Davis at Orbitz in 2006, stored and graphed any number you sent it. In 2011 Etsy released StatsD, a small daemon that took counts and timings over UDP and flushed them to Graphite every 10 seconds, so tracking something new needed no setup at all.

Graphite names were dotted strings, so slicing by anything that wasn’t in the name was hard. At SoundCloud, hitting those limits with hundreds of services, a colleague’s 2012 pet project became Prometheus: a multi-dimensional data model with labels such as instance, service, endpoint and method, a query language to slice by them, and metrics pulled over HTTP. Google’s SRE book later noted how closely Prometheus resembles Borgmon.

Methods for choosing what to measure got names too. Brendan Gregg published the USE method in 2012, Tom Wilkie created RED in 2015 for microservices, and Google’s Site Reliability Engineering book (2016) set out the four golden signals and warned against designing around averages: a service with a 100 ms mean can easily have 1% of requests taking 5 seconds. By 2026 the clouds had converged on the open formats; in June CloudWatch began accepting OpenTelemetry metrics and answering PromQL queries.

Why it’s built this way

Labels instead of names with everything baked in
One metric name with key-value labels can be summed, filtered or grouped by any label at query time. The cost is that each new combination of label values is a new series, so the label set has to stay bounded.
Pull over HTTP
Prometheus scrapes targets so that it can tell reliably when one is down, so extra servers (even a laptop) can scrape the same targets, and so a person can open a target’s metrics in a browser. Its authors call pull “slightly better” and keep the Pushgateway for batch jobs that can’t be scraped.
Fire-and-forget UDP
StatsD used UDP because it’s fast and the application doesn’t care whether StatsD is up: if packets are lost the graphs go a bit wonky, but the site stays up.
Buckets, not percentiles, on the client
A percentile computed inside each process can’t be combined with others. Counting requests into latency buckets lets you add buckets from every server and then estimate the percentile for the whole service.

Timeline

  1. 2003

    Google creates Borg, and soon Borgmon to monitor it

  2. 2006

    Chris Davis writes Graphite at Orbitz

  3. 2011

    Etsy releases StatsD

  4. 2012

    Prometheus starts at SoundCloud; Brendan Gregg publishes the USE method

  5. 2015

    Tom Wilkie creates the RED method

  6. 2016

    Google’s SRE book names the four golden signals

  7. 2026

    CloudWatch adds native OpenTelemetry metrics with PromQL in June

Sources20

Improve this page on GitHub