Cloud Parts

Logs

Timestamped records of what a program did, written as it runs and shipped somewhere you can search them.

11 technologies on this page ↓
Live model
What you’re watchingTwo services write DEBUG (pink), INFO (blue) and ERROR (orange) lines. The node’s log agent drops DEBUG, ships the rest to the log store, and also sends errors to an alert.

How it works

A log is a stream of events, each a line with a timestamp, a severity level and a message, such as 2026-10-06T09:14:03Z ERROR payment declined order=9124. The program writes them as things happen. In containers and on platforms such as Kubernetes, Lambda or Cloud Run, the convention is to write to stdout and stderr and let the platform capture the stream; Kubernetes keeps each container’s output in files on the node and rotates them at 10 MiB by default. An agent such as Fluent Bit, Vector or the OpenTelemetry Collector reads those streams, adds metadata (pod, host, service), filters out what you don’t want and ships the rest to a central store: CloudWatch Logs, Cloud Logging, Azure Monitor Logs, Loki, Elasticsearch, OpenSearch or a ClickHouse-based stack such as ClickStack. There you search every machine at once. Levels decide how much detail you keep: most services log INFO and above in production and turn DEBUG on only while chasing a bug. The bill follows volume. Hosted stores charge per GB ingested (CloudWatch Logs’ pricing examples and Cloud Logging’s price list both use $0.50 per GB, Cloud Logging after a free 50 GiB per project each month) plus storage, so retention periods, dropping noisy lines and sampling repetitive ones matter as much as what you log.

Think of it as

A ship’s logbook. The crew writes down each event with the time as it happens. Nobody reads it while the sea is calm, but after a storm it’s the first thing everyone opens.

Reach for it when

  • Knowing exactly what happened to one request, job or user, with the details
  • Errors and their context: the inputs, the IDs and the branch the code took
  • Audit trails: who changed what, and when
  • Questions you couldn’t predict a metric for in advance

Think twice when

  • Counting things over time; a metric is far cheaper than counting log lines
  • Following one request across many services by hand; put a trace or correlation ID in every line
  • Passwords, tokens, card numbers and personal data; mask them before they leave the process
  • DEBUG on everything in production; volume, and the bill, grow with traffic

A real example

A checkout service writes about 200 log lines a second at INFO, around 1.5 KB each. That’s roughly 26 GB a day, close to 780 GB a month, or about $390 a month to ingest at $0.50 per GB before storage. After an incident someone leaves DEBUG on and the volume jumps, say, tenfold. A filter in the node’s Fluent Bit that drops DEBUG in production, and a 30-day retention period on the log group (CloudWatch keeps logs forever unless you set one), bring the bill back down.

More on logs

Narrower ideas that build on this one.

Technologies

Specific products, and what each does differently.

Also used for logs

Products of other kinds that are often used this way, grouped by what they are.

Observability →

Amazon CloudWatchAWSAWS’s built-in monitoring service: metrics from every AWS service, logs, alarms and dashboards, and now traces, OpenTelemetry and PromQL, billed per metric, per GB and per alarm.Better StackCommercialUptime checks, on-call paging and status pages, plus logs, traces, metrics and errors in a ClickHouse-based store, sold as one SaaS billed per responder and per GB.ClickStackOpen sourceClickHouse’s open-source observability stack: an OpenTelemetry collector, ClickHouse for storage and the HyperDX UI for search, dashboards and alerts on logs, traces, metrics and session replays.DatadogCommercialA hosted monitoring platform: an agent on every host ships metrics, logs and traces to Datadog’s SaaS, which bills per host, per GB and per metric.OpenTelemetryOpen sourceThe vendor-neutral standard for producing and shipping traces, metrics and logs, with SDKs for most languages and a Collector to route the data.

What each provider calls it

AWS
CloudWatch Logs (Logs Insights for queries, Live Tail)
Google Cloud
Cloud Logging (Log Router and log buckets)
Azure
Azure Monitor Logs (Log Analytics workspaces, queried with KQL)
Open source & others
Stores: Grafana Loki, Elasticsearch, OpenSearch, ClickStack; agents: Fluent Bit, Fluentd, Vector, Logstash, OpenTelemetry Collector; hosted: Datadog, Better Stack

Log levels#

Syslog’s eight severities and the six ranges in OpenTelemetry’s log data model, side by side. Most logging libraries use names close to the left column.

LevelSyslog severity (RFC 5424)OpenTelemetry SeverityNumberUse it for
TRACEnone1 to 4Step-by-step detail; off almost everywhere
DEBUG7 Debug5 to 8Detail for developers; off in production unless you’re chasing a bug
INFO6 Informational, 5 Notice9 to 12Normal events worth keeping: started, order placed, job finished
WARN4 Warning13 to 16Something odd the code handled: a retry, a fallback, a slow call
ERROR3 Error17 to 20A request or job failed
FATAL2 Critical, 1 Alert, 0 Emergency21 to 24The process or system can’t go on
  • Rows line up by meaning; the two scales don’t map one to one. In syslog a lower number is more severe, in OpenTelemetry a higher one.

In code · bash

# the container writes to stdout; the platform keeps the stream
kubectl logs deployment/api --since=10m --follow

# the same lines after an agent shipped them to CloudWatch Logs
aws logs tail /ecs/api --since 10m --follow --filter-pattern '{ $.level = "error" }'

# keep 30 days instead of forever
aws logs put-retention-policy --log-group-name /ecs/api --retention-in-days 30

Often confused with

Structured loggingMetricsDistributed tracingObservabilityLogs, metrics or traces?
Where it came from2001–2022

Programs have always printed what they were doing, but on early Unix each daemon kept its own files in its own format. In the 1980s Eric Allman, while building Sendmail at Berkeley, wrote syslog: a program sends a short message tagged with a facility (which part of the system is talking) and a severity, and a syslog daemon writes it locally or relays it to a central host. It became the standard on nearly every Unix-like system and network device. It was only written down in 2001, when RFC 3164 described the BSD syslog protocol as it had been observed in the wild; RFC 5424 replaced it in 2009 with a stricter format, structured data fields and a required TLS transport.

Cloud platforms changed where logs live. When servers come and go, files on their disks go with them. Heroku’s Twelve-Factor App, presented by Adam Wiggins around 2011, stated the new rule: an app never manages log files; each process writes an unbuffered event stream to stdout, and the platform collects, routes and archives it. Agents grew up to do that routing, among them Logstash, Fluentd and, from 2014, Fluent Bit, a lightweight sibling of Fluentd built at Treasure Data for small devices. AWS added CloudWatch Logs the same year. In 2018 Grafana Labs introduced Loki, which borrowed Prometheus’ labels to make storing logs cheap.

The latest shift is a shared model. OpenTelemetry declared its log data model stable in 2022, so a log record has the same fields (timestamp, severity, body, attributes, and the trace and span IDs of the request it belongs to) whatever language or agent produced it.

Why it’s built this way

A severity on every message
Syslog tagged each message with one of eight severities, from Emergency to Debug. One setting could then decide what to keep and where to send it, without changing the program.
Write to stdout and let the platform route it
If the app doesn’t know or care where its logs go, the same code works on a laptop, where you watch the stream in a terminal, and in production, where it’s collated and sent to an indexing system.
Index labels, not the text
Loki indexes only a small set of labels per log stream and keeps the lines themselves compressed in object storage. Its authors chose this so that logs would be cheap enough that nobody is asked to log less, and so that the same labels as Prometheus metrics lead from an alert to the right logs.

Timeline

  1. 2001

    RFC 3164 documents the BSD syslog protocol as observed

  2. 2009

    RFC 5424 standardises syslog and obsoletes RFC 3164

  3. 2011

    The Twelve-Factor App: treat logs as event streams

  4. 2014

    Fluent Bit begins at Treasure Data; CloudWatch Logs launches in July

  5. 2018

    Grafana Labs introduces Loki

  6. 2022

    OpenTelemetry declares its log data model stable

Sources19

Improve this page on GitHub