Cloud Parts

Apache Kafka

The open-source distributed log that most event streaming is built on, or built to be compatible with.

What it is
Event streams
Used for
Pub/sub,Message queues
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
5 October 2026
Website Source code

What it is

Kafka stores each topic as a set of partitions, and each partition as a series of segment files on a broker’s disk. A producer picks the partition by hashing the record’s key, so every event for one customer lands, in order, in the same partition. Each partition is copied to several brokers: one leader takes the writes and the followers replicate them. Records stay until the retention limit (seven days by default) or, with log compaction, until a newer record with the same key replaces them.

Since Kafka 4.0 (March 2025) a cluster runs entirely in KRaft mode: a few controller nodes keep the cluster’s metadata in their own Raft-replicated log, and the separate ZooKeeper ensemble that older versions needed is gone. Around the core sit Kafka Connect, a framework of ready-made source and sink connectors for databases, object stores and warehouses, and Kafka Streams, a Java library for stateful stream processing. Share groups (“Queues for Kafka”), production-ready since 4.2, let many consumers split the records of the same partitions and acknowledge them one by one, like a queue.

Why teams pick it

  • Sequential disk writes and batching give very high throughput on ordinary hardware
  • Consumers can rewind to any retained offset, so a new service or a fixed bug can reprocess history
  • The biggest ecosystem in streaming: Connect, Streams, Flink, Spark, and a protocol that Amazon MSK, Confluent Cloud, Redpanda and Azure Event Hubs all speak
  • Log compaction turns a topic into a durable “latest value per key” changelog

Watch out for

  • Running it yourself is real work: partition counts, replication, rebalancing, disk sizing and upgrades; many teams pay for a managed service instead
  • The partition count caps parallelism (one consumer per partition in a group), and adding partitions remaps keys and breaks per-key ordering
  • No per-message routing, delays or priorities; that logic lives in your consumers
  • Long retention on broker disks is expensive unless you set up tiered storage (production-ready since 3.9) with a remote-storage plugin

Alternatives

What all event streams have in common →
Amazon Kinesis Data StreamsAWSAWS’s serverless event stream, split into shards that each keep records in order.Apache FlussOpen sourceStreaming storage built for Flink: event logs and updatable tables in columnar Arrow format, with key lookups, and older data copied into Paimon, Iceberg, Hudi or Lance tables.Apache PulsarOpen sourceA multi-tenant messaging and streaming platform from Yahoo that keeps brokers stateless and stores messages in Apache BookKeeper, so one topic can serve as a log, a pub-sub feed or a work queue.Cloudflare K2BetaCommercialCloudflare’s serverless event stream: a durable log stored in R2 that Workers and HTTP clients append to, read through independent subscriptions that lease out batches of records.RedpandaCommercialA Kafka-compatible streaming broker rewritten in C++ as one binary, with no JVM and no ZooKeeper.WarpStreamCommercialA Kafka-compatible stream that keeps no data on disks: stateless Agents write straight to your object storage bucket.

How it fits with other technologies

What uses Apache Kafka

Create a topic, produce keyed records and read them as a consumer group · bash

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic orders --partitions 6 --replication-factor 3

# type lines like  customer-42:{"total": 99}
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
  --property parse.key=true --property key.separator=:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \
  --group billing --from-beginning

# where is each partition’s offset, and how far behind is the group?
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing

See it in a real stack

Open lakehouse on Iceberg →
Live model
What you’re watchingServices publish order events to Kafka. Spark reads them in micro-batches and commits them to an Iceberg table whose files live on S3. Dashboard queries run on Trino, which reads the same table.
Event-driven order processingCheckout publishes one event to Kafka; payments, inventory and email each consume it at their own pace.Apache KafkaPostgreSQLAWS Lambda
Where it came from2010–2025

Around 2010, LinkedIn had two kinds of data that didn’t fit together. Databases held users and connections, but a much bigger stream of events (page views, searches, clicks, server metrics) had to reach Hadoop for offline analysis and live systems like search and newsfeed at the same time. The messaging systems of the day, such as ActiveMQ, focused on rich delivery guarantees, were hard to scale and slowed down badly when unread messages piled up. Log collectors like Facebook’s Scribe moved data in bulk to Hadoop but couldn’t feed anything in real time.

Jay Kreps, Neha Narkhede and Jun Rao built Kafka to do both: take in billions of events a day and let any number of systems read them, in real time or hours later. They described it in a 2011 paper at the NetDB workshop. Kreps named it after Franz Kafka because it was “a system optimized for writing” and he liked the author’s work. LinkedIn open-sourced it in January 2011, and in 2014 the three creators left to found Confluent around it.

Kafka changed how companies think about data movement. Instead of point-to-point pipelines between every pair of systems, many now publish events once to a central log, and every other system subscribes.

Why it’s built this way

An append-only log on disk instead of a message queue
Writing to the end of a file and reading it in order is the fastest thing a disk does. Keeping data for days, rather than deleting it on delivery, lets slow consumers, batch jobs and new systems replay history.
Consumers track their own position
The broker doesn’t record which messages each consumer has seen, so serving one more consumer costs almost nothing, and a consumer can rewind just by changing its offset.
Partitions as the unit of scale and order
Splitting a topic into partitions spreads it over many machines. Ordering is promised only within a partition, which is enough for “all events for one user, in order” without a global bottleneck.
Lean on the operating system
Kafka relies on the OS page cache instead of its own cache, and on sendfile to copy data from disk to network without passing through the application, which is a large part of why it’s fast.

Timeline

  1. 2010

    Built at LinkedIn to move activity data and metrics

  2. 2011

    Open-sourced in January; NetDB paper; enters the Apache Incubator in July

  3. 2012

    Graduates to a top-level Apache project in October

  4. 2014

    Its creators found Confluent

  5. 2017

    Exactly-once delivery and transactions arrive in 0.11; version 1.0 follows in November

  6. 2019

    KIP-500 proposes replacing ZooKeeper with Kafka’s own Raft-based quorum (KRaft)

  7. 2025

    Kafka 4.0 runs without ZooKeeper entirely and previews queues (KIP-932)

Sources21 · facts checked 5 October 2026

Improve this page on GitHub