Apache Pinot

A real-time distributed OLAP database from LinkedIn, built to answer analytics queries from end users in milliseconds at hundreds of thousands of queries per second, over data that arrived from Kafka seconds ago.

Apache · Top-level project

What it is: Real-time analytics databases
Type: Open source
Made by: Apache Software Foundation
Foundation: The Apache Software Foundation, top-level project, joined October 2018
License: Apache 2.0
Facts checked: 24 October 2026

What it is

Pinot stores each table as segments: immutable columnar files, each with its own dictionaries and indexes. A realtime table reads a stream such as Kafka or Kinesis; each server writes incoming rows into an in-memory consuming segment that is queryable at once, then commits it as a finished segment and uploads it to a deep store such as S3, GCS or HDFS. An offline table is filled by batch jobs that build segments and push them to the deep store. A hybrid table pairs the two under one name. Servers host segments on local disk and run queries against them; brokers send each query to the servers holding the relevant segments and merge the results; a controller assigns segments to servers and runs the cluster through Apache Helix, which keeps cluster state in ZooKeeper. Optional minions do background work such as converting, merging and purging segments.

Speed comes from choosing indexes per column: inverted, sorted, range, Bloom filter, text, JSON, geospatial, timestamp and vector indexes, and the star-tree index, which pre-aggregates over several dimension columns so that a matching GROUP BY reads a handful of rows instead of millions. Pinot assumes data is immutable but supports upserts by primary key on streaming tables, as long as the stream is partitioned by that key; the map of primary keys to their latest rows lives in server heap memory. Simple scatter-gather queries run on the default single-stage engine; the multi-stage engine, production-ready since Pinot 1.0 (September 2023) and switched on per query, adds distributed joins, window functions and subqueries. Pinot is free under Apache 2.0; StarTree, founded by two of its creators, sells a managed Pinot service with extra features.

Why teams pick it

Built for user-facing analytics: many small aggregation queries per second, each answered in milliseconds, such as LinkedIn’s “Who viewed your profile” and Uber Eats’ restaurant dashboards
Rows from Kafka or Kinesis are queryable within seconds, without a separate batch step
Indexes are configured per column, and the star-tree index keeps latency flat for known aggregation patterns as data grows
Upserts and deduplication by primary key on streaming tables, including partial upserts that change only some columns

Watch out for

Not a transactional database: no ACID transactions, and the docs advise against it when you need frequent single-row changes
Upsert tables have strict rules: the stream must be partitioned by primary key, all segments of a partition must sit on one server, and the key map uses heap memory
Several JVM services to run (controller, broker, server, minion) plus ZooKeeper and a deep store
Fast queries depend on picking indexes ahead of time; star-tree indexes help only queries that match their configured dimensions and aggregations, and joins need the multi-stage engine

Good for

Sub-second queries on fresh eventsRows from Kafka or Kinesis are queryable in a consuming segment at once, and per-column and star-tree indexes keep user-facing queries in milliseconds.

Specs

Compare with alternatives →

Data model: Tables of immutable columnar segments: realtime, offline or hybrid
Query language: SQL; the multi-stage engine adds joins, window functions and subqueries
Secondary indexes: Chosen per column: inverted, sorted, range, Bloom, text, JSON, geospatial, vector, star-tree
Transactions: None; upserts by primary key on streaming tables
How it scales: Brokers scatter queries to the servers holding segments; a controller assigns segments through Helix
Storage format: Columnar segments with per-column dictionaries and indexes; deep store on S3, GCS or HDFS
Ways to run it: Self-hosted (Helm chart on Kubernetes), or managed by StarTree

Alternatives

What all real-time analytics databases have in common →Compare side by side →

Apache DruidOpen sourceAlso good for sub-second queries on fresh eventsAn open-source real-time analytics database that makes Kafka and Kinesis events queryable within seconds and serves fast slice-and-dice queries to dashboards and APIs.

ClickHouseOpen sourceAlso good for sub-second queries on fresh eventsAn open-source column-oriented database for real-time analytics, fast enough to serve dashboards over billions of rows.

StarRocksOpen sourceAlso good for sub-second queries on fresh eventsAn open-source MPP analytical database with a vectorized engine that serves fast joins and real-time updates on its own tables, and also queries Iceberg, Hudi, Delta Lake and Hive tables in place.

Apache DorisOpen sourceAlso good for sub-second queries on fresh eventsAn open-source MPP analytical database, born at Baidu as Palo, that serves sub-second queries on fresh, updatable data through the MySQL protocol and also queries Iceberg, Hudi, Paimon and Hive tables in place.

Azure Data ExplorerAzureMicrosoft’s managed analytics database for logs, telemetry and time series, known internally as Kusto and queried with KQL; it began as the back end of Azure Monitor’s Log Analytics and Application Insights.

TinybirdCommercialAlso good for sub-second queries on fresh eventsA hosted ClickHouse platform that turns SQL queries into authenticated REST API endpoints, for analytics inside your product.

CrateDBOpen sourceAlso good for sub-second queries on fresh eventsA distributed SQL database built on Lucene: every column is indexed and stored column by column, so one cluster answers aggregations, full-text, geo and vector queries over fresh machine data.

How it fits with other technologies

What Apache Pinot uses

A realtime table that reads a Kafka topic, from the Pinot quickstart · json

{
  "tableName": "transcript",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "timestampInEpoch",
    "timeType": "MILLISECONDS",
    "schemaName": "transcript",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.topic.name": "transcript-topic",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory",
      "stream.kafka.broker.list": "kafka:9092",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.segment.size": "50M",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
    }
  },
  "metadata": { "customConfigs": {} }
}

See it in a real stack

No architecture uses Apache Pinot yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2015–2026

LinkedIn shows analytics to its members and customers, such as who viewed a profile, Talent Insights and company analytics, and runs internal tools such as the XLNT A/B testing platform and ThirdEye metrics monitoring. Each of these sends many small slice-and-dice queries over billions of rows, and each needs an answer fast enough to render a page. LinkedIn built Pinot for this, ran it internally for over two years, and open-sourced it under Apache 2.0 in June 2015, by then backing more than 25 analytics products.

Pinot entered the Apache Incubator in October 2018 and published its first Apache release, 0.1.0, in March 2019. Uber adopted it for real-time dashboards, including Uber Eats’ Restaurant Manager, and Microsoft Teams piloted it on Azure. It graduated to a top-level Apache project in July 2021, the year Kishore Gopalakrishna and Xiang Fu, two of its creators at LinkedIn, launched StarTree to sell it as a managed service.

Pinot started with its own SQL-like language, PQL, and simple filter-and-aggregate queries. Pinot 1.0 in September 2023 made the multi-stage engine, with distributed joins and window functions, ready for production, and later releases kept adding upsert, time-series and vector features.

Why it’s built this way

Immutable columnar segments with per-column indexes: User-facing queries must stay fast at high concurrency, so Pinot reads as little data as possible: dictionaries, inverted, sorted and range indexes let a query touch only matching rows.
Pre-aggregate with a star-tree index: Fully pre-computing every combination explodes storage, while scanning raw rows is too slow; the star-tree keeps partial aggregates for chosen dimensions, which puts an upper bound on the rows a matching query reads.
Real-time and offline tables under one name: Fresh data comes from a stream and history from batch jobs; a hybrid table lets queries span both, with a time boundary deciding which side answers.
Cluster management through Apache Helix: Helix, also from LinkedIn, already handled partition assignment, replicas and failover on top of ZooKeeper, so Pinot didn’t have to build its own.

Timeline

2015
LinkedIn open-sources Pinot in June after more than two years in production
2018
Enters the Apache Incubator in October
2019
First Apache release, 0.1.0, in March
2021
Graduates to a top-level Apache project in July; StarTree launches
2023
Pinot 1.0: the multi-stage engine for joins is ready for production
2026
Pinot 1.5.0 in April; 1.6.0 release candidates in October

Sources9 · facts checked 24 October 2026

Improve this page on GitHub