Cloud Parts

Distributed tracing

Follow one request through every service it touches, as a tree of timed spans that share one trace ID.

7 technologies on this page ↓
Live model
What you’re watchingA request travels from the gateway through checkout to payments and the answer comes back the same way, carrying one trace ID in its traceparent header. Each service also reports its own span (green) to the collector, which joins them by trace ID into one tree.

How it works

When a request enters the system, the first service creates a trace ID and a span: a record of one unit of work with a name, a start time, a duration, a status and attributes. Every call it makes carries the trace ID and its own span ID onward, usually in the W3C traceparent HTTP header, or in message headers when the call goes through a queue. The next service reads the header, starts a child span whose parent is the caller’s span, and passes its own span ID on. Each service sends its finished spans to a collector, and the tracing backend joins them by trace ID into a tree, drawn as a waterfall: the whole request at the top, and every downstream call and database query nested underneath with its own bar. That shows at a glance where the time went and which hop failed, which metrics (no per-request detail) and logs (scattered across services) can’t show on their own. Instrumentation libraries for common HTTP servers, clients and database drivers, such as OpenTelemetry’s zero-code instrumentation, create most spans without changes to your code. Tracing every request costs a lot, so systems sample. Head sampling decides at the start, often by keeping a fixed share of trace IDs, such as 5%; it’s cheap, but it can’t know in advance which requests will fail. Tail sampling decides once a trace is complete, usually in a collector, and can keep every trace with an error or high latency, at the cost of holding all spans until the decision.

Think of it as

A parcel’s tracking history. One tracking number follows it through every depot, and each depot adds a scan with the time, so you can see exactly where it sat for two days.

Reach for it when

  • Requests that cross several services, queues or functions
  • Finding which hop made a request slow, or where it failed
  • Seeing the real dependency map of a system from its traffic
  • Jumping from a bad metric or an error log to an example request

Think twice when

  • One process and one database; a profiler and good logs tell you more
  • Counting and trending; derive metrics instead of querying millions of traces
  • Keeping every trace at high traffic without a budget; sample

A real example

Checkout p95 jumps from 400 ms to 2.1 s. The latency histogram says it’s slow; a trace of one slow request says why. The gateway span lasts 2.1 s and the checkout span inside it almost as long, but most of that is a payments call that spends 1.8 s waiting on a fraud-check service deployed ten minutes earlier. The collector’s tail sampler kept that trace because it was slow, and its trace_id also finds the matching log lines in all four services.

More on distributed tracing

Narrower ideas that build on this one.

Used for distributed tracing

Distributed tracing is something products do, not a kind of product. These are often used for it, grouped by what they are.

Observability →

Amazon CloudWatchAWSAWS’s built-in monitoring service: metrics from every AWS service, logs, alarms and dashboards, and now traces, OpenTelemetry and PromQL, billed per metric, per GB and per alarm.Better StackCommercialUptime checks, on-call paging and status pages, plus logs, traces, metrics and errors in a ClickHouse-based store, sold as one SaaS billed per responder and per GB.ClickStackOpen sourceClickHouse’s open-source observability stack: an OpenTelemetry collector, ClickHouse for storage and the HyperDX UI for search, dashboards and alerts on logs, traces, metrics and session replays.DatadogCommercialA hosted monitoring platform: an agent on every host ships metrics, logs and traces to Datadog’s SaaS, which bills per host, per GB and per metric.OpenTelemetryOpen sourceThe vendor-neutral standard for producing and shipping traces, metrics and logs, with SDKs for most languages and a Collector to route the data.

What each provider calls it

AWS
AWS X-Ray, with traces in the CloudWatch console (Transaction Search, Application Signals)
Google Cloud
Cloud Trace
Azure
Application Insights (Azure Monitor)
Open source & others
OpenTelemetry for instrumentation and collection; backends: Jaeger, Zipkin, ClickStack

One trace as a waterfall#

The slow checkout from the example, span by span. All six spans share one trace ID; each row’s parent is the span that made the call.

SpanServiceParent spanStarts atDuration
POST /checkoutgatewaynone (the root)0 ms2,104 ms
PlaceOrdercheckoutPOST /checkout6 ms2,090 ms
SELECT cart_itemscheckout, in PostgreSQLPlaceOrder9 ms14 ms
POST /chargepaymentsPlaceOrder30 ms1,860 ms
POST /scorefraud-checkPOST /charge41 ms1,800 ms
INSERT orderscheckout, in PostgreSQLPlaceOrder1,900 ms11 ms
  • Read it as nested bars: POST /score sits inside POST /charge, which sits inside PlaceOrder. The 1,800 ms bar is where to look.

In code · text

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  |                                |                flags: 01 = sampled
             |  |                                parent-id: the caller’s span (16 hex)
             |  trace-id: the same on every hop of this request (32 hex)
             version

gateway  -> checkout   traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
checkout -> payments   traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-b7ad6b7169203331-01
                       same trace-id, new parent-id: payments’ span is a child of checkout’s

Often confused with

Correlation IDsLogsMetricsObservabilityLogs, metrics or traces?
Where it came from2010–2020

Research systems such as Pinpoint and Magpie explored tracing requests across components, but the design most tools copy is Google’s Dapper, described in a 2010 paper after more than two years in production. At Google one request could fan out over many services written by different teams, in different languages, on thousands of machines. Dapper modelled each request as a tree of spans sharing a trace ID, kept the trace context in thread-local storage and carried it through Google’s shared RPC and callback libraries so applications needed almost no changes, and sampled heavily: one trace in 1,024 was enough for most uses.

Twitter built Zipkin during its first Hack Week as a basic version of the Dapper paper for its Thrift services and open-sourced it in June 2012. Uber created Jaeger in 2015; it entered the CNCF in 2017 and graduated in 2019. The bottleneck was instrumentation: every library had to be instrumented for every backend, and two vendor-neutral APIs competed, OpenTracing (in the CNCF from October 2016) and Google’s OpenCensus (open-sourced in January 2018). In 2019 they merged into OpenTelemetry.

Propagation needed a standard too. Each tool used its own headers, Zipkin’s b3 and X-B3-* among them, so a trace broke wherever two vendors met. The W3C Trace Context specification, a Recommendation since February 2020 (revised November 2021), defined the traceparent and tracestate headers, which OpenTelemetry uses by default.

Why it’s built this way

A trace as a tree of spans
Each span records its own ID, its parent’s ID and the shared trace ID, so a backend can rebuild the causal tree from spans that arrive in any order from any host, and still see which call waited on which.
Instrument shared libraries, not applications
Almost all of Google’s services used one RPC framework and one callback library. Putting the tracing there made it nearly transparent: even web search could be traced without application-level annotations.
Sample, and decide early
Dapper’s authors found that tracing every request slowed latency-sensitive services, and that one request in thousands was enough. Deciding at the root and passing the decision on in the trace flags keeps every hop of a sampled trace and none of an unsampled one.
One header every vendor understands
With vendor-specific headers, traces collected by different tools couldn’t be correlated and intermediaries dropped the metadata. A W3C standard gives clouds, proxies and libraries one format to forward.

Timeline

  1. 2010

    Google publishes the Dapper paper

  2. 2012

    Twitter open-sources Zipkin in June

  3. 2015

    Uber creates Jaeger

  4. 2016

    OpenTracing joins the CNCF

  5. 2018

    Google open-sources OpenCensus

  6. 2019

    OpenTracing and OpenCensus merge into OpenTelemetry; Jaeger graduates from the CNCF

  7. 2020

    W3C Trace Context becomes a Recommendation in February

Sources13

Improve this page on GitHub