Joins and windows over unbounded streams
Continuous computations that remember things between events, such as counts per window, joins between two streams or running totals per key.
What it asks of a system
Filtering or reshaping one event at a time needs no memory. Counting clicks per five-minute window, joining orders with payments, or keeping a running balance per account does: the processor has to keep state, partitioned by key, for as long as the computation needs it. Events also arrive late and out of order, so a window over event time has to wait for a watermark that says no older events are expected before it emits a result.
To tell whether a system really handles it, look at where it keeps state (memory, local RocksDB, remote storage) and how large that state can grow, how it snapshots state so a crash resumes without losing or double-counting events, and whether it reads event time and watermarks or only the wall clock. Exactly-once results need both consistent snapshots and a source that can replay from a position, such as a Kafka offset, plus a sink that can take transactional or idempotent writes.
Good at it · 9
Compare side by sideStream processing
ArroyoOpen sourceSQL windows and stream joins keep state that is checkpointed with source offsets to object storage, with exactly-once output to Kafka.A stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.
FelderaOpen sourceKeeps whole SQL programs, joins, aggregates and window functions included, up to date with work proportional to each change.An incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.
ksqlDBCommercialPersistent SQL queries run as Kafka Streams apps with local state and changelogs, covering joins and windowed aggregations over topics.Confluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.
PathwayCommercialJoins, group-bys and windows run on a Differential Dataflow engine that updates results incrementally, and late events revise earlier outputs.A Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.
Quix StreamsOpen sourcePer-key RocksDB state backed by changelog topics, event-time windows and joins, and exactly-once Kafka output, all in pure Python.A pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.Sources
- Flink docs: Stateful stream processing (keyed state, checkpoints, exactly once)
- Flink docs: Timely stream processing (event time, processing time, watermarks)
- Flink docs: Windows (tumbling, sliding, session)
- Kafka docs: Kafka Streams architecture (tasks, local state, changelog topics)
Why each technology is listed comes from the sources on its own page.