cloudparts

Joins and windows over unbounded streams

Continuous computations that remember things between events, such as counts per window, joins between two streams or running totals per key.

What it asks of a system

Filtering or reshaping one event at a time needs no memory. Counting clicks per five-minute window, joining orders with payments, or keeping a running balance per account does: the processor has to keep state, partitioned by key, for as long as the computation needs it. Events also arrive late and out of order, so a window over event time has to wait for a watermark that says no older events are expected before it emits a result.

To tell whether a system really handles it, look at where it keeps state (memory, local RocksDB, remote storage) and how large that state can grow, how it snapshots state so a crash resumes without losing or double-counting events, and whether it reads event time and watermarks or only the wall clock. Exactly-once results need both consistent snapshots and a source that can replay from a position, such as a Kafka offset, plus a sink that can take transactional or idempotent writes.

Good at it · 9

Compare side by side

Stream processing

Apache FlinkOpen sourceKeyed state in memory, RocksDB or ForSt is checkpointed with source offsets, and event-time windows close on watermarks.The most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.ArroyoOpen sourceSQL windows and stream joins keep state that is checkpointed with source offsets to object storage, with exactly-once output to Kafka.A stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.FelderaOpen sourceKeeps whole SQL programs, joins, aggregates and window functions included, up to date with work proportional to each change.An incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.Kafka StreamsOpen sourceEach task keeps RocksDB state backed by a compacted changelog topic, and exactly_once_v2 makes Kafka-to-Kafka processing exactly once.A Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.ksqlDBCommercialPersistent SQL queries run as Kafka Streams apps with local state and changelogs, covering joins and windowed aggregations over topics.Confluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialJoins, group-bys and windows run on a Differential Dataflow engine that updates results incrementally, and late events revise earlier outputs.A Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Quix StreamsOpen sourcePer-key RocksDB state backed by changelog topics, event-time windows and joins, and exactly-once Kafka output, all in pure Python.A pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.RisingWaveOpen sourceStreaming joins and aggregations keep their state in an LSM store on object storage, with a consistent checkpoint every second by default.A streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.

Sources

Why each technology is listed comes from the sources on its own page.