Durable, replayable event streams
An ordered, replicated log of events that many independent consumers read at their own pace, and can read again from any point.
What it asks of a system
In an event stream, producers append events to a log and nothing is deleted when a consumer reads them. Each consumer keeps its own position, so a billing service, a search indexer and a nightly batch job can all read the same events, and a new service or a fixed bug can rewind and reprocess history. Events with the same key go to the same partition and are read in the order they were written; the log is replicated across machines so a failed broker loses nothing.
To tell whether a system really handles it, check how long it can keep data and what that costs (broker disks or object storage), whether it orders events per key or per partition, what happens to ordering when you add partitions, and what delivery guarantee consumers get: at least once is common, exactly once needs transactions or idempotent writes. Compatibility with the Kafka protocol also matters, since most connectors and stream processors speak it.
Good at it · 8
Compare side by sideEvent streams
Amazon Kinesis Data StreamsAWSShards keep records in order per partition key for up to 365 days, and any consumer can re-read from a position inside that window.AWS’s serverless event stream, split into shards that each keep records in order.
Apache FlussOpen sourceEach bucket is a replicated log, and primary-key tables emit a full changelog that Flink jobs can read from any point.Streaming storage built for Flink: event logs and updatable tables in columnar Arrow format, with key lookups, and older data copied into Paimon, Iceberg, Hudi or Lance tables.
Apache KafkaOpen sourceTopics are replicated, partitioned logs kept for a set time, and each consumer group tracks its own offset, so any reader can rewind and replay.The open-source distributed log that most event streaming is built on, or built to be compatible with.
Apache PulsarOpen sourceTopics are replicated BookKeeper ledgers with per-subscription cursors, and tiered storage keeps old backlog readable from object storage.A multi-tenant messaging and streaming platform from Yahoo that keeps brokers stateless and stores messages in Apache BookKeeper, so one topic can serve as a log, a pub-sub feed or a work queue.
Cloudflare K2BetaCommercialRecords stay in an R2-backed log for 1 hour to 30 days whether read or not, and each subscription can start from the earliest one.Cloudflare’s serverless event stream: a durable log stored in R2 that Workers and HTTP clients append to, read through independent subscriptions that lease out batches of records.
RedpandaCommercialA Kafka-compatible log with a Raft group per partition, so Kafka clients, replay by offset and tiered retention work as they do on Kafka.A Kafka-compatible streaming broker rewritten in C++ as one binary, with no JVM and no ZooKeeper.
WarpStreamCommercialKafka-compatible topics stored as files in your own bucket, so long retention and replay cost object-storage prices, not broker disks.A Kafka-compatible stream that keeps no data on disks: stateless Agents write straight to your object storage bucket.
Sources
- Kafka docs: Introduction (durable storage, read as often as needed, per-partition order, replication)
- Kafka docs: Design (consumer position, partitions and consumer groups)
- Confluent blog: Exactly-once semantics are possible, here’s how Kafka does it (June 2017)
Why each technology is listed comes from the sources on its own page.