Cloud Parts

Event streams

An append-only log of events that many readers replay at their own pace.

10 technologies on this page ↓
Live model
What you’re watchingRecords stay in the log after they’re read. Three consumer groups move through the same records at different speeds; the ML job falls behind, then catches up in batches.

How it works

Producers append records to the end of a log that’s kept for a retention period: hours, days or forever. Reading removes nothing. Each consumer group tracks its own position, called an offset, so a real-time alerting job, a dashboard and a nightly batch can all read the same data independently. Logs are split into partitions for scale, and order is guaranteed within a partition.

Think of it as

A DVR recording of live TV. You can watch live, pause, or rewind to this morning, and your roommate can be at a different point in the same recording.

Reach for it when

  • High-volume telemetry, clickstreams and IoT readings
  • Several independent consumers of the same events
  • Reprocessing history after a bug fix or to build a new view

Think twice when

  • Simple job distribution where each task runs once; a queue is simpler
  • Low volume, where running a stream platform is more work than it saves

A real example

Thousands of well sensors publish pressure readings to a readings topic. Alerting reads each one within a second, the dashboard updates every few seconds, and the ML team replays the whole week on Sunday to retrain a model.

More on event streams

Narrower ideas that build on this one.

Technologies

Specific products, and what each does differently.

Amazon Kinesis Data StreamsAWSAWS’s serverless event stream, split into shards that each keep records in order.Apache FlussOpen sourceStreaming storage built for Flink: event logs and updatable tables in columnar Arrow format, with key lookups, and older data copied into Paimon, Iceberg, Hudi or Lance tables.Apache KafkaOpen sourceThe open-source distributed log that most event streaming is built on, or built to be compatible with.Apache PulsarOpen sourceA multi-tenant messaging and streaming platform from Yahoo that keeps brokers stateless and stores messages in Apache BookKeeper, so one topic can serve as a log, a pub-sub feed or a work queue.Cloudflare K2BetaCommercialCloudflare’s serverless event stream: a durable log stored in R2 that Workers and HTTP clients append to, read through independent subscriptions that lease out batches of records.RedpandaCommercialA Kafka-compatible streaming broker rewritten in C++ as one binary, with no JVM and no ZooKeeper.WarpStreamCommercialA Kafka-compatible stream that keeps no data on disks: stateless Agents write straight to your object storage bucket.PgQueOpen sourceSkype’s PgQ engine repackaged as one SQL file: a zero-bloat event log inside any Postgres 14+.Mainly Message queuesEstuaryCommercialA managed data-movement platform that captures changes from databases and SaaS sources into collections kept as JSON files in your own bucket, then keeps warehouses and other systems in sync, in real time or on a schedule.Mainly ETL & ELT pipelines

Also used for event streams

Products of other kinds that are often used this way, grouped by what they are.

What each provider calls it

AWS
Kinesis Data Streams, MSK (Kafka)
Google Cloud
Managed Service for Apache Kafka, Pub/Sub
Azure
Event Hubs
Open source & others
Apache Kafka, Redpanda (source-available, BSL), Apache Pulsar; WarpStream (Kafka API on object storage); Apache Fluss (streaming storage with updatable tables, for Flink)

See it in a real stack

Architectures that use event streams

Open lakehouse on IcebergKafka streams events into Spark, which writes Iceberg tables on S3 that Trino queries for dashboards.Apache KafkaApache SparkApache IcebergAmazon S3TrinoEvent-driven order processingCheckout publishes one event to Kafka; payments, inventory and email each consume it at their own pace.Apache KafkaPostgreSQLAWS LambdaIoT telemetry pipelineDevices stream readings into Kinesis; Lambda writes them to TimescaleDB for Grafana, and raw data is archived to S3.Amazon Kinesis Data StreamsAWS LambdaTimescaleDBAmazon S3Grafana

In code · python

from kafka import KafkaProducer, KafkaConsumer

producer = KafkaProducer(bootstrap_servers="broker:9092")
producer.send("readings", key=b"well-42", value=b'{"psi": 812.4}')

# Each group_id keeps its own offset in the same log
consumer = KafkaConsumer("readings", group_id="alerts",
                         bootstrap_servers="broker:9092",
                         auto_offset_reset="earliest")
for record in consumer:
    check_thresholds(record.offset, record.value)

Worth exploring

Article· September 2025Understanding Apache FlussJack VanlightlyA long, detailed walk through Fluss internals, reverse-engineered from the code: log tables, primary-key tables that emit changelogs, and tiering to lakehouse tables. The author works at Confluent, which runs Flink but not Fluss, and says so up front.
All resources →

Often confused with

Message queuesPub/subTime-series databasesEvent busesStream processingQueue, stream or pub/sub?
Where it came from2010–2026

Before event streams, companies moved data between systems with one-off pipelines. At LinkedIn around 2010, Jay Kreps recalled, “we built jerry-rigged piping between systems and applications on an as needed basis,” and the complexity meant “the data was always unreliable.” Off-the-shelf brokers such as ActiveMQ were hard to scale, and dumps into Hadoop arrived in batches rather than in real time.

In 2010 LinkedIn decided to build one system for streams of data: Apache Kafka, open-sourced in 2011. The team called it “the global commit log thingy”: an ordered, append-only record that every system could write to and read from at its own pace. Amazon Kinesis offered the same model as a managed cloud service in 2013, and Apache Pulsar, which keeps storage separate from its brokers, became a top-level Apache project in 2018. Retention grew from hours to months, so a stream became a durable record that new systems could replay from the start.

Streams and queues have since converged from both sides: RabbitMQ added streams in 2021, and Kafka made queue-style share groups production-ready in 2026.

Why it’s built this way

Reading doesn’t delete
Records stay for a retention period whoever has read them, so a dashboard, an alerting job and a nightly batch can all consume the same data, and a bug fix can be followed by a replay.
Each consumer keeps its own position
The log doesn’t track delivery per message, so adding readers is cheap, and a reader can rewind just by moving its offset.
Partitions for scale, order within each
Splitting a log across machines lets it grow; promising order only within a partition (one device, one user) avoids a global bottleneck.
Sequential writes to disk
Appending to the end of a file is the fastest thing a disk does, which lets a log absorb very high write rates on ordinary hardware.

Timeline

  1. 2010

    LinkedIn starts building Kafka to replace ad hoc pipelines

  2. 2011

    Kafka is open-sourced

  3. 2013

    Amazon Kinesis brings managed streams to the cloud

  4. 2018

    Apache Pulsar becomes a top-level Apache project

  5. 2020

    Kinesis retention can be extended to 365 days

  6. 2021

    RabbitMQ 3.9 adds streams

  7. 2026

    Kafka 4.2 makes queue-style share groups production-ready

Sources13

Improve this page on GitHub