Cloud Parts

Stream processing

Queries that never finish: they read events as they arrive and keep results, counts and joins up to date.

16 technologies on this page ↓
Live model
What you’re watchingClicks are split by user across two partitions, and each partition feeds one task of the job. Every event updates that task’s per-user state; only now and then does a window close and send a result on. All of each user’s events go to the same task, so its count lives in one place.

How it works

A stream processor runs a query that never ends: it reads events from a stream, such as a Kafka topic, and emits results while new events keep arriving. Stateless steps, like filtering or reshaping, look at one event at a time. Stateful steps remember things between events: a join holds events until their match arrives, and an aggregation keeps a running count per key. Aggregations are cut into windows: tumbling (back-to-back minutes), hopping (overlapping, say five minutes long and starting every minute) or session (closed after a quiet gap). Windows usually follow event time, when something happened, rather than processing time, when it arrived, and a watermark says how far event time has got, so the processor knows when a window is complete. The work is split by key across parallel tasks, each keeping its state locally, in memory or in RocksDB. The processor saves that state together with its input positions (Flink in periodic checkpoints, Kafka Streams in changelog topics), so after a crash it resumes without losing or double-counting events. Exactly-once output also needs a sink that takes part, through transactions or idempotent writes.

Think of it as

A tally clerk at a stadium’s gates. Instead of counting everyone from the CCTV recording the next morning, they keep a running count per gate on a clipboard, call out a total every fifteen minutes, and photocopy the clipboard now and then so a stand-in can carry on from that point if they leave.

Reach for it when

  • Live metrics, counters and dashboards that have to be seconds old, not hours
  • Fraud, anomaly and alerting rules over a window of recent events per key
  • Joining or enriching one stream with another, or with a table kept current by change data capture
  • Streaming ETL: cleaning, reshaping and routing events on their way into a warehouse or lakehouse

Think twice when

  • An answer that’s an hour old is fine; a scheduled batch query is easier to write, test and rerun
  • Questions you can’t plan in advance; a stream job only answers the query it was started with
  • Each event just triggers one action, such as sending an email; a queue and workers are simpler
  • Nobody can look after it: long-running stateful jobs need checkpoints, upgrades and watching for lag and growing state

A real example

Card payments arrive on a payments topic, keyed by card number. A Flink job counts declined payments per card in one-minute tumbling windows by event time, with a watermark ten seconds behind the newest event so stragglers still count. More than five declines in a minute sends an alert. When a worker dies, the job restarts from its last checkpoint and rereads the topic from the offsets stored in it, so no decline is lost or counted twice.

Technologies

Specific products, and what each does differently.

Libraries inside your app

Stream processing as code in your own service, with no cluster to run.

Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

What each provider calls it

AWS
Managed Service for Apache Flink (named Kinesis Data Analytics until 2023)
Google Cloud
Dataflow (Apache Beam)
Azure
Stream Analytics
Open source & others
Apache Flink (StreamFusion, pre-1.0, runs Flink SQL natively in Rust), Kafka Streams, Spark Structured Streaming, Arroyo; streaming databases and incremental views: Feldera (MIT core, paid Enterprise features), RisingWave (Apache 2.0; premium features need a licence key) and Materialize (BSL); ksqlDB (Confluent Community License); Redpanda Connect for stateless pipelines; Python: Quix Streams, Pathway (BSL), Bytewax (now community-maintained); Rust: Timely and Differential Dataflow, Streamling (FSL)

See it in a real stack

Architectures that use stream processing

Open lakehouse on IcebergKafka streams events into Spark, which writes Iceberg tables on S3 that Trino queries for dashboards.Apache KafkaApache SparkApache IcebergAmazon S3Trino

In code · sql

-- Flink SQL: page views per minute, by event time, waiting 5 s for stragglers
CREATE TABLE clicks (
  user_id STRING,
  page    STRING,
  ts      TIMESTAMP(3),
  WATERMARK FOR ts AS ts - INTERVAL '5' SECOND
) WITH (
  'connector' = 'kafka',
  'topic' = 'clicks',
  'properties.bootstrap.servers' = 'broker:9092',
  'properties.group.id' = 'page-views',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'json'
);

-- one row per page per window, emitted once the watermark passes window_end
SELECT window_start, window_end, page, COUNT(*) AS views
FROM TUMBLE(TABLE clicks, DESCRIPTOR(ts), INTERVAL '1' MINUTES)
GROUP BY window_start, window_end, page;

Worth exploring

Article· October 2024Stream Processing With Apache Flink: book notesChris Shumaker, Maker ConsultingOne engineer’s chapter-by-chapter notes on the O’Reilly book by Fabian Hueske and Vasiliki Kalavri, read to understand what owning a Flink job in production involves: state, time, scale and performance. A quick way to decide whether the book is worth reading in full.
All resources →

Often confused with

Event streamsMessage queuesPub/subWorkflow orchestrationQuery enginesETL & ELT pipelines
Where it came from2011–2026

For years, “real time” meant queues and hand-written workers. At BackType, Nathan Marz recalled, most of that code dealt with moving and serialising messages rather than the actual logic, so he built Storm to hide the plumbing and released it in September 2011, after Twitter bought BackType. Exact results still came from batch jobs, so many teams ran the same logic twice: fast in a stream job, correct in Hadoop. In 2014 Jay Kreps called maintaining code for the same result in two distributed systems “exactly as painful as it seems like it would be”, and proposed replaying a retained log through the stream job instead.

Google’s MillWheel (2013) and Dataflow Model (2015) papers showed how a stream job could be correct on its own, with exactly-once state, event time, windows and watermarks. Apache Flink, which grew out of the Stratosphere research project and became a top-level Apache project in December 2014, paired those ideas with consistent snapshots of all its state. Kafka Streams (2016) put similar machinery into a library.

Today most engines also speak SQL, and the clouds sell them as services: Google’s Dataflow runs Apache Beam pipelines, Azure has Stream Analytics, and in 2023 AWS renamed Kinesis Data Analytics to Managed Service for Apache Flink, then retired its older SQL engine.

Why it’s built this way

Keep state inside the processor
Looking up a database for every event is slow and hammers the database. Keeping counts, windows and join buffers in local memory or RocksDB, split by key across tasks, keeps each step fast.
Event time and watermarks, not arrival time
Events arrive late and out of order, and replays arrive all at once. Grouping by when things happened, and using watermarks to decide when a window is done, gives the same answer live and on replay.
Snapshot state and input positions together
Writing every update somewhere durable would be too slow. Saving state and offsets as one consistent checkpoint lets a job rewind to that point after a crash, so nothing is lost or counted twice.
Replay instead of a second batch system
If the input is kept in a log, fixing a bug means rerunning the stream job from the start, not keeping a separate batch pipeline in step with it.

Timeline

  1. 2011

    Storm is open-sourced at Strange Loop, after Twitter buys BackType

  2. 2013

    Google’s MillWheel paper describes low watermarks and exactly-once processing

  3. 2014

    Flink becomes a top-level Apache project; Kreps questions the Lambda Architecture

  4. 2015

    The Dataflow Model paper sets out event time, windows and watermarks

  5. 2016

    Kafka Streams ships in Kafka 0.10; Spark 2.0 previews Structured Streaming

  6. 2023

    Kinesis Data Analytics becomes Amazon Managed Service for Apache Flink

  7. 2026

    AWS deletes the last Kinesis Data Analytics for SQL applications

Sources30

Improve this page on GitHub