Self-hosted observability on the LGTM stack
Loki for logs, Grafana for dashboards, Tempo for traces and Mimir for metrics, fed by one OpenTelemetry pipeline and storing everything in object storage.
Click any box to open its page. Open it in the playground to rewire it, change the traffic or take parts down.
Grafana Labs calls its four open-source projects the LGTM stack, after the code-review shorthand “looks good to me”: Loki for logs, Grafana to look at them, Tempo for traces and Mimir for metrics. The three backends share one design: a stateless distributor takes writes, recent data is kept close at hand for queries, and everything older is written as files to an object storage bucket, which is the only place data is kept for the long term. Services are instrumented once with OpenTelemetry, and a collector, either the OpenTelemetry Collector or Grafana Alloy, sends each signal to its store. Grafana then queries each store in its own language, PromQL, LogQL and TraceQL, and links them through shared labels and trace IDs. It suits teams that want metrics, logs and traces in one place but would rather run the backends themselves and pay mainly for object storage.
How it flows
- 1
Instrument once, collect in one place
Services use OpenTelemetry SDKs and send OTLP to a collector: the OpenTelemetry Collector, or Grafana Alloy, Grafana Labs’ distribution of it with built-in Prometheus pipelines that also scrapes
/metricsendpoints. The collector sends each signal to its backend: logs to Loki’s/otlpendpoint, metrics to Mimir’s/otlpendpoint or by Prometheus remote write, and spans to Tempo over OTLP.ServicesOpenTelemetry - 2
Loki indexes labels and stores log lines as chunks
A few resource attributes, such as
service.nameandk8s.namespace.name, become Loki labels; other attributes are kept as structured metadata. The distributor hashes each stream to ingesters, which compress lines into chunks and flush them, with a small TSDB index of the labels, to the bucket. Searching a word means picking streams by label and scanning their chunks. - 3
Mimir writes through Kafka and keeps Prometheus blocks
In Mimir 3’s ingest storage architecture, distributors validate samples and shard series across Kafka partitions; a write is acknowledged once Kafka has replicated it. Each ingester consumes one partition into a per-tenant Prometheus TSDB, cuts a block every two hours and uploads it. The compactor merges and deduplicates blocks, and store-gateways serve them to queries.
- 4
Tempo turns spans into Parquet blocks
In Tempo 3’s microservices mode, distributors write spans, sharded by trace ID, to Kafka. Block-builders consume them into Apache Parquet blocks in the bucket, and live-stores serve the most recent spans. An optional metrics-generator derives rate, error and duration metrics and a service graph from the spans and remote-writes them to Mimir, so every traced service gets request-rate, error and latency metrics.
- 5
Grafana queries each store and links them
Grafana has built-in Loki, Tempo and Prometheus data sources; Mimir is added as a Prometheus data source. A panel sends PromQL to Mimir, LogQL to Loki or TraceQL to Tempo, and each backend splits the query in its query frontend and reads recent data plus files from the bucket. Because all three use the same labels, Grafana can move from a latency spike to the traces and logs of the same service and time. Alert rules run in Grafana Alerting or in the Mimir and Loki rulers.
Why this shape works
- Object storage holds all data at rest, so long retention costs bucket prices rather than replicated SSDs, and none of the three needs a separate database
- One OpenTelemetry pipeline feeds all three stores, and shared labels such as
service.namemake jumping between metrics, traces and logs work without lookup tables - Each signal scales on its own: log volume, active series and span rate grow at different speeds, and each backend’s write and read paths scale separately
- All four are open source (Grafana, Loki, Tempo and Mimir under AGPLv3) and are also sold as Grafana Cloud, so a team can move between self-hosted and hosted without changing queries or dashboards
Where it strains
- Three distributed databases to run, each with its own components, hash rings and caches; Mimir 3 and Tempo 3 also want a Kafka-compatible cluster for production, so this is a platform team’s job, not a side task
- Labels are the index in both Loki and Mimir: a label with unbounded values (user ID, request ID, pod name in some setups) multiplies streams and series and the memory and cost behind them
- Loki has no full-text index, so a broad search over days of logs scans a lot of chunks; teams used to Elasticsearch need to narrow by label first
- AGPLv3 obliges anyone who modifies these projects and offers them over a network to publish the changes, and some companies forbid AGPL software outright
- Correlation only works if the labels really match: the same service name and environment on logs, metrics and spans, and trace IDs written into log lines
Common swaps
One Kubernetes cluster and a few weeks of metrics? A single Prometheus server is simpler than Mimir; add Mimir when you need long retention, high availability or many clusters in one view.
Rather run one database than three? ClickStack keeps logs, metrics and traces in ClickHouse and queries them with SQL.
Need full-text search over logs, such as security investigations across every field? Elasticsearch or OpenSearch index every word, at a higher storage cost.
On premises or another cloud? Point all three at MinIO, Google Cloud Storage or Azure Blob Storage instead of S3.
Don’t want to run any of it? A hosted platform such as Datadog or Better Stack takes the same OpenTelemetry data.
Built from
OpenTelemetryOpen sourceObservabilityThe vendor-neutral standard for producing and shipping traces, metrics and logs, with SDKs for most languages and a Collector to route the data.
Grafana LokiOpen sourceLogsA log store that indexes only a few labels per stream, keeps the log lines as compressed chunks in object storage, and greps them at query time with LogQL.
Grafana MimirOpen sourceMetricsLong-term, multi-tenant storage for Prometheus metrics: it takes remote write and OTLP, keeps Prometheus TSDB blocks in object storage and answers PromQL across them.
Grafana TempoOpen sourceDistributed tracingA tracing backend that keeps every span as Parquet files in object storage, with no database to run, and finds traces by trace ID or with TraceQL.
Amazon S3AWSObject storageAWS’s object storage, and the de facto API most object stores copy.
GrafanaOpen sourceObservabilityDashboards and alerts over data that stays where it is: Prometheus, SQL databases, cloud monitoring and more.Architectures with parts in common
Sources
- Grafana Labs blog: Open source at Grafana Labs in 2022 (the LGTM stack, “looks good to me”)
- Grafana Alloy docs: Introduction (OpenTelemetry Collector distribution with Prometheus pipelines; metrics, logs, traces, profiles)
- Grafana Loki docs: Ingesting logs to Loki using OpenTelemetry Collector (/otlp, default labels, structured metadata)
- Grafana Loki docs: Architecture (distributor, ingester, index and chunks in one object store)
- Grafana Mimir docs: Ingest storage architecture (Kafka partitions, ingesters, compactor)
- Grafana Mimir docs: Configure the OpenTelemetry Collector to write metrics into Mimir (native OTLP)
- Grafana Tempo docs: Architecture (distributor, Kafka, block-builder, live-store, metrics-generator, Parquet)
- Grafana docs: Data sources (built-in Prometheus, Loki and Tempo)
- GitHub: grafana/docker-otel-lgtm (Collector, Prometheus, Tempo, Loki, Pyroscope and Grafana in one image, for development)