Cloud Parts

Timely Dataflow

A Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

What it is
Stream processing
Type
Open source
Made by
Frank McSherry and contributors (TimelyDataflow project)
License
MIT (since the first commit in 2014)
Facts checked
6 October 2026
Website Source code

What it is

Timely Dataflow is a Rust crate, not a server. Your program describes a dataflow graph of operators, and timely::execute starts a number of worker threads, optionally in several processes on different machines; every worker builds the same graph and handles its own share of the data. An exchange step routes each record to a worker by key, through memory between threads or over TCP between processes. Every batch of records carries a logical timestamp, and timestamps can be partially ordered: inside a loop, a stream’s time becomes a pair of (outer time, iteration count), which is how a dataflow can contain cycles. Workers constantly share progress information. Each operator holds capabilities, the times at which it may still send data, and the system turns them into a frontier for every input: the earliest times that can still arrive. When the frontier passes a time, the operator knows that time is complete and can emit a final result; a probe tells the driving program when a round of input has been fully processed.

Its sister library, Differential Dataflow, sits on top and supplies the higher-level operators. A collection is a stream of (data, time, diff) updates, and map, join, reduce and iterate respond to each change by doing only the work that change requires, keeping indexed state called arrangements that several operators can share. Both are MIT-licensed libraries with nothing to buy and nothing to deploy beyond your own binary. They also leave out what a stream processing service provides: SQL, connectors, persistence and recovery from failed workers. Materialize built a source-available SQL database on top of both, and Pathway’s engine runs on modified copies of them.

Why teams pick it

  • Loops and incremental updates in one dataflow: iterative algorithms such as graph reachability keep their answers current as the input changes
  • Exact progress tracking: frontiers come from what upstream operators can still send, so an operator knows when a time is complete instead of guessing
  • The same program runs on one thread or on many processes across machines, chosen with command-line flags
  • Rust with no garbage collector, and proven in production as the compute layer of Materialize

Watch out for

  • A library for Rust programmers: no SQL, no connectors to Kafka or databases, no scheduler, UI or deployment tooling
  • No fault tolerance: unlike Spark or Flink it doesn’t survive a failed worker, so checkpoints, persistence and restarts are yours to build
  • The number of workers and processes is set when the program starts, and every worker must build the same dataflow
  • Still pre-1.0 after more than ten years: 55 releases on crates.io (0.31.0 in July 2026), with breaking API changes, and older blog examples may need tweaks to compile

Alternatives

What all stream processing have in common →
Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.

How it fits with other technologies

Run the hello example on two worker threads, then on two processes · bash

# in your own project: cargo add timely differential-dataflow
git clone https://github.com/TimelyDataflow/timely-dataflow && cd timely-dataflow

# one process, two worker threads; `exchange` routes each number to a worker by value
cargo run --example hello -- -w2
# worker 0:   hello 0
# worker 1:   hello 1
# ...

# two processes with two workers each (run each line in its own terminal);
# without -h they connect over localhost
cargo run --example hello -- -w2 -n2 -p0
cargo run --example hello -- -w2 -n2 -p1

# across machines: a file of host:port lines, the same on every process
#   cargo run --example hello -- -w2 -h hostfile.txt -n4 -p0   # on host0, then -p1 on host1, ...

Timely, Differential, and what’s built on them#

Timely Dataflow is the bottom layer. Most people meet it through something built on top.

Timely Dataflow

MIT, Rust library

Workers, data exchange, logical timestamps, loops and progress tracking. Operators are closures you write; state lives in memory inside them.

Differential Dataflow

MIT, Rust library

Collections of (data, time, diff) updates with incremental join, reduce, distinct and iterate, and shared indexed state (arrangements). Compiles down to timely dataflow.

Materialize

BSL 1.1

A PostgreSQL-compatible database that keeps SQL views up to date from Kafka and database replication streams. Founded in 2019 by Frank McSherry and Arjun Narayan; it depends on the timely and differential-dataflow crates and adds SQL, persistence in object storage and replication.

Pathway

BSL 1.1

A Python stream processing framework whose Rust engine is built on forked and modified copies of both libraries, kept in its own repository.

See it in a real stack

No architecture uses Timely Dataflow yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2013–2026

Around 2012, data processing was split across separate systems: batch processors for throughput, stream processors for low latency, and specialised graph systems for iterative algorithms. Applications that needed all three stitched several platforms together. At Microsoft Research, Derek Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham and Martín Abadi built Naiad to do all of it in one engine. Its model, timely dataflow, attached logical timestamps to data so that cyclic, incremental computations could coordinate cheaply; the paper appeared at SOSP in November 2013, after a CIDR paper earlier that year on differential computation.

Naiad was written in C#, and Microsoft Research’s leadership ended the project in 2014. McSherry started again in Rust: the first commit is from December 2014, and the first crate was published in May 2015. His stated aims were to learn Rust and to build reusable infrastructure for data-parallel research, without fighting C#’s performance. Differential Dataflow followed in 2015 as its own library.

In January 2019 McSherry and Arjun Narayan founded Materialize to turn the two libraries into a streaming SQL database, first released in February 2020. The libraries stayed MIT-licensed and are still developed in the open, and Pathway later built its engine on forks of them.

Why it’s built this way

Logical timestamps that can be partially ordered
A timestamp like (input round, loop iteration) lets the system tell apart work from different rounds and different iterations at once. That is what makes loops inside a streaming computation possible.
Progress tracking instead of global synchronisation
Operators report which times they may still produce, and everyone else learns when a time is complete. Naiad’s authors called this an efficient, lightweight coordination mechanism: no stop-the-world steps between rounds.
Every worker runs the whole dataflow on a shard of the data
Workers are identical and data moves only at explicit exchange points, so the same program scales from one thread to many machines without a central scheduler.
No built-in fault tolerance
McSherry argued in 2015 that fault tolerance costs real overhead (writing to disk, stateless operators) and is not worth it for computations that finish in seconds. Systems that need it, such as Materialize, add their own.

Timeline

  1. 2013

    Differential dataflow paper at CIDR; Naiad paper at SOSP

  2. 2014

    Naiad ends at Microsoft; McSherry starts timely dataflow in Rust

  3. 2015

    First timely crate published; Differential Dataflow library begins

  4. 2019

    McSherry and Arjun Narayan found Materialize on top of both

  5. 2023

    Pathway publishes its engine, built on forks of both libraries

  6. 2026

    Timely 0.31.0 released in July

Sources13 · facts checked 6 October 2026

Improve this page on GitHub