High-cardinality time series
A steady flood of timestamped points from many sources, each tagged with labels, that you query by time range and aggregate.
What it asks of a system
Metrics, sensor readings and market ticks arrive as timestamped values, mostly in time order, and are rarely changed afterwards. Each series is named by its labels (host, region, device, symbol), and every distinct combination of label values is a separate series. Cardinality is the number of those series. A label like a user ID or a container ID can turn thousands of series into millions, and many systems slow down or run out of memory long before they run out of disk.
A system built for this partitions data by time, compresses each series well (timestamps and values change slowly), drops or downsamples old data on a schedule, and can answer “average per minute over the last day, per host” without reading raw points one by one. To tell whether one handles your load, test with your real number of series rather than your number of points, and look at what happens to memory and query speed as series churn, for example when every deploy creates new pod names.
Good at it · 17
Compare side by side
Time-series databases
Amazon TimestreamAWSTimestream for InfluxDB runs managed InfluxDB, with line protocol and Telegraf, and InfluxDB 3 clusters keep data as Parquet in S3.AWS’s time-series databases: a serverless engine now closed to new customers, and managed InfluxDB 2 and 3.Apache IoTDBOpen sourceWrites go to a WAL-backed memtable and flush to TsFiles, while late readings go to separate files merged later by compaction.An Apache time-series database for industrial IoT that stores everything in its own open file format, TsFile, and lets you model devices as a tree of paths or as SQL tables.
TimechoDBCommercialThe IoTDB engine: writes go to a WAL-backed memtable and flush to TsFiles, and late readings go to separate files merged later.The commercial edition of Apache IoTDB from the company its creators founded: the same engine plus dual-active pairs, one-way data-diode sync, a Workbench console and vendor support, licensed per machine with activation codes.ArcOpen sourceTakes InfluxDB line protocol, buffers rows in memory and flushes them every few seconds as Parquet files partitioned by hour.A young single-binary time-series database in Go that writes incoming data as Parquet files and queries them with DuckDB SQL.
GreptimeDBOpen sourceTakes Prometheus remote write, OpenTelemetry and line protocol, and flushes each table region as Parquet to object storage.A Rust database that stores metrics, logs and traces as tables on object storage, queried with SQL or PromQL.
InfluxDBOpen sourceLine protocol writes go to a WAL and an in-memory buffer, then to Parquet on object storage, without the old series-index cardinality limits.The best-known time-series database, fed by line protocol and Telegraf; now in its third generation, a Rust rewrite that keeps data as Parquet files on object storage.
KairosDBOpen sourcePoints go into Cassandra rows per metric, tag set and three-week window, so ingest inherits Cassandra’s write throughput and scaling.A Java time-series server that stores metrics with tags in Apache Cassandra; it began in 2013 as a rewrite of OpenTSDB and is now kept going by a single maintainer.
kdb+CommercialA tickerplant takes the feed, logs it and publishes it to a real-time database that keeps today’s rows in RAM.A columnar time-series database that is also an interpreter for q, an array language, used for market data across trading and risk.
QuestDBOpen sourceMany clients write through a WAL into time-partitioned column files, with out-of-order rows and deduplication handled on write.A columnar time-series database in Java, C++ and Rust that ingests millions of rows a second and answers time-series SQL, with old partitions kept as Parquet.
TDengineOpen sourceEach device gets its own table, kept whole in one vnode, so a write is an append; supertables let one query cover every device.A clustered time-series database for IoT and industrial data that gives every device its own table, grouped under “supertables” with tags.
TimescaleDBOpen sourceHypertables split rows into time chunks: inserts land in the newest chunk, queries skip old ones, and retention drops whole chunks.A PostgreSQL extension that turns ordinary tables into time-partitioned, compressed time-series tables.ValkeyTimeSeriesPreviewOpen sourceSeries are keys of compressed chunks, an inverted label index finds matching series, and batched inserts run in parallel.A community module, written in Rust, that adds an in-memory time-series data type to Valkey with the TS.* commands of RedisTimeSeries plus Prometheus-style label queries, joins and anomaly detection.
Workers Analytics EngineCommercialAny blob or index can be a user ID or URL with no cardinality charge, and adaptive sampling keeps fast-writing keys cheap.Cloudflare’s time-series analytics store for Workers: a non-blocking writeDataPoint() call to record events with unlimited cardinality, adaptive sampling to keep it cheap, and a SQL API to query them.Real-time analytics databases
Azure Data ExplorerAzureIngests millions of events per second into indexed, compressed extents, and KQL has built-in time-series, anomaly and forecasting functions.Microsoft’s managed analytics database for logs, telemetry and time series, known internally as Kusto and queried with KQL; it began as the back end of Azure Monitor’s Log Analytics and Application Insights.
CrateDBOpen sourceTables partitioned by a generated time column skip whole partitions in queries and drop old ones with a single `DELETE`.A distributed SQL database built on Lucene: every column is indexed and stored column by column, so one cluster answers aggregations, full-text, geo and vector queries over fresh machine data.
Sources
- Prometheus docs: Metric and label naming (each label combination is a new time series)
- Prometheus docs: Storage
- Tiger Data docs: Understand hypertables (chunks by time range)
- Tiger Data docs: add_retention_policy()
Why each technology is listed comes from the sources on its own page.