Cloud Parts

Logs, metrics or traces?

DimensionLogsMetricsDistributed tracing
In one lineTimestamped records of eventsNumbers aggregated over timeOne request’s path through the services, as timed spans
One unit of dataA text line or JSON object per eventA sample of a labelled time series, such as a counter, gauge or histogramA span: trace ID, parent span ID, name, start, duration, status
Best at answeringWhat exactly happened to this request, and why?How much, how fast, and is it getting worse?Where did the time go, and which hop failed?
Cost grows withTraffic and verbosity: GB ingested, stored and scannedThe number of series (label combinations), not trafficTraffic times spans per request, unless you sample
SamplingDrop DEBUG and noisy repeats; keep every errorNot needed: the data is already aggregatedThe norm: head (a fixed share) or tail (keep errors and slow traces)
AlertingPossible through log-based metrics or queries; slower and costlierThe usual basis for alerts and SLOsRarely on traces themselves; on metrics derived from spans
Breaks whenLines are free text, or share no ID, so ten services’ logs can’t be joinedA label has unbounded values, or someone averages percentilesOne service in the path doesn’t pass the header on, so the trace splits
StandardsSyslog (RFC 5424); OpenTelemetry log data modelPrometheus exposition format; StatsD; OpenTelemetry metricsW3C Trace Context; OpenTelemetry traces
Joins to the others bytrace_id and span_id fields on each lineExemplars: sample values that carry a trace IDThe trace ID, which finds the logs; span data can produce rate, error and duration metrics
Where it’s storedCloudWatch Logs, Cloud Logging, Azure Monitor Logs, Loki, Elasticsearch, OpenSearch, ClickStackPrometheus, CloudWatch metrics, Cloud Monitoring, Azure Monitor MetricsX-Ray, Cloud Trace, Application Insights, Jaeger, Zipkin, ClickStack

Rule of thumb

You need all three, in this order: metrics to notice (alert on rate, errors and duration), traces to find the slow or failing hop, and logs for the details of that one request. Make them join from the start: the same service name everywhere and the trace ID in every log line.

Sources