Cloud Parts

Real-time analytics databases

Columnar databases that make streaming events queryable within seconds and answer many small analytical queries at once, fast enough to sit behind an app.

11 technologies on this page ↓
Live model
What you’re watchingEvents land on a stream, and ingest tasks add them to the store within seconds. Meanwhile customer dashboards fire many small queries; each comes back quickly because it reads only indexed, pre-aggregated columns.

How it works

A real-time analytics database stores events column by column, like a warehouse, but it is built for a different workload: many small queries every second, from an app or an API, on data that is seconds old. Ingestion never stops. The database reads a stream such as a Kafka topic, or takes HTTP posts, and indexes new rows in memory, where queries can already see them. It then writes them out as immutable columnar files, called segments or parts, sorted by time and a key, and merges small files into bigger ones in the background. Queries stay fast because they read little. Indexes skip most of the data: Druid keeps a bitmap for every value of a column, ClickHouse a sparse index on the sort key, Pinot a star-tree of pre-aggregated values. Pre-aggregation shrinks it further: rows can be rolled up at ingestion (one row per minute per page instead of one per click), or a materialized view keeps running totals updated on every insert. Hot data sits on local disk or in memory, and the full copy usually lives in object storage. Compared with a warehouse, which loads in batches and spends a lot of compute on each big query from a few analysts, the goal here is freshness in seconds and hundreds or thousands of concurrent queries that each return in milliseconds, mostly against one wide event table; joins and row updates are weaker. Compared with a time-series database, which is organised around series (a metric plus tags) and retention, these systems expect high-cardinality columns such as user IDs and URLs and arbitrary filters and group-bys across many columns, although the two overlap.

Think of it as

The scoreboard in a stadium rather than the league’s archive. It changes within seconds of each play and thousands of people read it at once, because it shows running tallies kept up all game, not the footage of every play. The archive, like a warehouse, has everything, for slower and deeper questions afterwards.

Reach for it when

  • Analytics inside your product: per-customer dashboards, usage meters and leaderboards that many users load at once
  • Data that must be queryable seconds after it happens: ad spend, fraud signals, game or device events arriving on a stream
  • Filters and group-bys over high-cardinality columns (user IDs, URLs, SKUs) answered in well under a second
  • APIs that return aggregates at hundreds or thousands of queries per second

Think twice when

  • Occasional analyst questions over years of history across many joined tables; a warehouse or lakehouse is cheaper and better at joins
  • Records that change one by one by primary key: Druid can only overwrite whole time ranges, and most others make updates costly (Pinot’s upserts are the exception)
  • Small data or few users: PostgreSQL with good indexes, or DuckDB, is far less to run
  • You need every raw event later but plan to roll up at ingestion; rollup throws the detail away

A real example

An email platform shows each of its 20,000 customers a live opens-and-clicks dashboard. Delivery events, about 50,000 a second, flow through Kafka into ClickHouse, and a materialized view keeps counts per customer, campaign and minute. A dashboard load runs four queries filtered on customer_id; each reads a few thousand pre-aggregated rows and returns in about 20 ms, even at 1,000 queries a second during a big send. Raw events stay in the base table for 30 days, for drill-downs.

Technologies

Specific products, and what each does differently.

Alibaba Cloud HologresCommercialAlibaba Cloud’s PostgreSQL-compatible real-time data warehouse, which takes streaming writes, answers analytical queries and serves key lookups from one copy of the data.Apache DorisOpen sourceAn open-source MPP analytical database, born at Baidu as Palo, that serves sub-second queries on fresh, updatable data through the MySQL protocol and also queries Iceberg, Hudi, Paimon and Hive tables in place.Apache DruidOpen sourceAn open-source real-time analytics database that makes Kafka and Kinesis events queryable within seconds and serves fast slice-and-dice queries to dashboards and APIs.FireboltCommercialA cloud data warehouse built for low-latency, high-concurrency queries behind apps and dashboards, also free to self-host as Firebolt Core.PivotOpen sourceA young open-source analytics server in Rust that speaks the PostgreSQL protocol and queries Iceberg and Delta Lake tables straight from object storage.StarRocksOpen sourceAn open-source MPP analytical database with a vectorized engine that serves fast joins and real-time updates on its own tables, and also queries Iceberg, Hudi, Delta Lake and Hive tables in place.TinybirdCommercialA hosted ClickHouse platform that turns SQL queries into authenticated REST API endpoints, for analytics inside your product.CedarDBCommercialA PostgreSQL-compatible database rebuilt from scratch at TU Munich, running transactions and analytics on one copy of the data on a single server.Mainly Relational databasesScramDBCommercialA brand-new, closed-source database that speaks the PostgreSQL protocol and promises transactions, analytics and vector search on one copy of the data; first builds shipped in summer 2026.Mainly Relational databasesClickHouseOpen sourceAn open-source column-oriented database for real-time analytics, fast enough to serve dashboards over billions of rows.Mainly Data warehousesDatabendOpen sourceA cloud data warehouse written in Rust that keeps every table as Parquet blocks and Git-like snapshots in object storage, so stateless compute clusters can start, scale and stop without moving data. Open core: Apache 2.0, with paid features under the Elastic License 2.0.Mainly Data warehouses

What each provider calls it

AWS
No dedicated service. Redshift streaming ingestion lands Kinesis or MSK data in materialized views; OpenSearch Service covers log and event analytics; otherwise people run ClickHouse, Druid or Pinot on EC2 or EKS, or use ClickHouse Cloud or Imply Polaris, which run on AWS
Google Cloud
No dedicated service either: BigQuery with the Storage Write API for streaming rows and BI Engine’s in-memory cache for dashboard queries; or ClickHouse Cloud or Tinybird in Google Cloud regions
Azure
Azure Data Explorer and Eventhouse in Microsoft Fabric Real-Time Intelligence (both queried with KQL): streaming ingestion, answers in milliseconds to seconds
Open source & others
ClickHouse, Apache Druid, Apache Pinot, StarRocks, Apache Doris; as services: ClickHouse Cloud, Tinybird (on ClickHouse), Imply (Druid), StarTree (Pinot), PhoenixAI (formerly CelerData, StarRocks), Firebolt, Alibaba Cloud Hologres; also CedarDB (one engine for transactions and analytics); early: Pivot (Postgres protocol over Iceberg and Delta tables), ScramDB (pre-release)

In code · sql

-- ClickHouse: raw events, sorted the way customers ask about them
CREATE TABLE events (
    ts DateTime,
    customer_id UInt32,
    page String,
    duration_ms UInt32
) ENGINE = MergeTree
ORDER BY (customer_id, ts);

-- A rollup kept current on every insert: one row per customer, page and minute
CREATE TABLE page_minutes (
    minute DateTime,
    customer_id UInt32,
    page String,
    views UInt64,
    total_ms UInt64
) ENGINE = SummingMergeTree
ORDER BY (customer_id, page, minute);

CREATE MATERIALIZED VIEW page_minutes_mv TO page_minutes AS
SELECT toStartOfMinute(ts) AS minute, customer_id, page,
       count() AS views, sum(duration_ms) AS total_ms
FROM events
GROUP BY minute, customer_id, page;

-- What one customer’s dashboard asks, many times a second
SELECT page, sum(views) AS views, sum(total_ms) / sum(views) AS avg_ms
FROM page_minutes
WHERE customer_id = 4812 AND minute >= now() - INTERVAL 1 HOUR
GROUP BY page
ORDER BY views DESC
LIMIT 10;

Worth exploring

WebsiteThe Umbra Database SystemThomas Neumann’s group, Technical University of MunichThe research site for Umbra, the successor to HyPer and the basis of CedarDB: a feature overview and the full publication list with PDFs, from the CIDR 2020 paper “Umbra: A Disk-Based System with In-Memory Performance” onwards. You can try queries in the browser or pull a Docker image, but its terms allow only evaluation, not production use.
All resources →

Benchmarks

Benchmark · ClickHouse Inc.ClickBenchHow fast analytical databases run 43 SQL queries over one 100-million-row table of web analytics events, on the same cloud machines.32 systems in this catalog · 7 caveats to read first
How to read a benchmark →

Often confused with

Data warehousesTime-series databasesStream processingQuery enginesEvent streams
Where it came from1993–2021

E. F. Codd coined “OLAP” in 1993, in a paper written for Arbor Software, maker of the Essbase cube database. Classic OLAP meant cubes: aggregates computed ahead of time from a warehouse and browsed by analysts. They were fast, but the data was hours old and only the planned dimensions could be sliced.

Internet companies with floods of events needed more. In 2011 Metamarkets, which sold ad analytics dashboards, found relational databases too slow for arbitrary drill-downs, while pre-computing aggregates into a key-value store took over 24 hours as dimensions grew. It built Druid, a column store with bitmap indexes that made streaming events queryable at once. LinkedIn built Pinot for member-facing features such as Who Viewed Profile, and Yandex open-sourced ClickHouse, born in web analytics, in 2016.

These systems moved analytics from a few analysts to end users: thousands of people at once, on data seconds old. Druid and Pinot became top-level Apache projects in 2019 and 2021, StarRocks and Apache Doris followed, and most are now also sold as managed services.

Why it’s built this way

Query raw or lightly rolled-up events, not pre-built cubes
Cubes limited users to reports someone had planned, and rebuilding them took hours. Metamarkets needed any combination of dimensions at sub-second latency, so Druid aggregated only lightly at ingestion and computed the rest at query time.
Make new rows queryable while still in memory
Druid’s real-time nodes kept an in-memory index that queries could read directly, then periodically persisted it to columnar files and handed finished segments to other nodes. Data was visible within seconds without waiting for a batch load.
Indexes that skip data instead of scanning it
Many concurrent queries are only affordable if each touches little data. Druid borrowed bitmap indexes from search engines, ClickHouse keeps a sparse index on the sort key, and Pinot’s star-tree index stores pre-aggregated results across several columns.
Keep the master copy in shared storage
Druid writes immutable segments to deep storage such as S3 or HDFS, and query servers only cache them. A failed server is replaced by downloading segments again, and even a whole cluster can be rebuilt.

Timeline

  1. 1993

    E. F. Codd coins the term OLAP

  2. 2011

    Metamarkets introduces Druid for real-time ad analytics

  3. 2012

    Druid is open-sourced in October

  4. 2016

    Yandex open-sources ClickHouse

  5. 2018

    Pinot, built at LinkedIn, enters the Apache Incubator

  6. 2019

    Druid becomes a top-level Apache project

  7. 2021

    Pinot graduates to a top-level Apache project

Sources26

Improve this page on GitHub