Cloud Parts

Kafka Streams

A Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.

What it is
Stream processing
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
5 October 2026
Website Source code

What it is

Kafka Streams is a library you add to a Java or Scala application, not a server you run. You describe a topology of steps (filter, map, join, count, windowed aggregate) with the Streams DSL or the lower-level Processor API, and every running copy of your app joins the same group through the Kafka brokers. Each partition of the input topics becomes a task, and tasks are spread over all instances and their threads; start another instance and some tasks move to it. The partition count is also the ceiling: an input topic with five partitions keeps at most five instances busy.

State lives next to the code. Each task keeps its state stores on local disk, in RocksDB by default, and Kafka Streams writes every change to a compacted changelog topic in Kafka. If an instance dies, its tasks restart elsewhere and rebuild their stores by replaying those changelogs; standby replicas, warm copies kept on other instances, cut that wait. Re-keying data for a join or groupBy goes through an internal repartition topic. With processing.guarantee set to exactly_once_v2 (brokers 2.5 or newer), Kafka transactions make each input record’s effect show up exactly once in the output topics and state stores. Input and output are always Kafka topics. It ships with Apache Kafka under Apache 2.0, so there is nothing to license or buy.

Why teams pick it

  • No cluster to run: it’s a JAR inside your service, deployed, scaled and monitored like any other app
  • Local RocksDB state, backed up to changelog topics, makes joins and windowed aggregations fast and recoverable
  • Exactly-once processing from Kafka to Kafka with one setting
  • Interactive queries let the app read its own state stores, for example to serve the current count per key over an API
  • Released with Apache Kafka itself; Kafka 4.2 added dead letter queues for bad records and a broker-side rebalance protocol

Watch out for

  • JVM only: the API is Java and Scala, so Python or Go teams need another engine
  • Kafka in, Kafka out: reaching databases or other systems means Kafka Connect or your own client code
  • Parallelism is capped by the input topics’ partition count
  • Big state means slow recovery: a moved task replays its whole changelog before it resumes, unless standby replicas are kept
  • Interactive queries only see the local instance; discovering the others and the RPC layer between them are yours to build

Alternatives

What all stream processing have in common →
Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

Run the bundled word-count app, feed it lines and watch the running counts · bash

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic streams-plaintext-input --partitions 1 --replication-factor 1
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic streams-wordcount-output --partitions 1 --replication-factor 1 \
  --config cleanup.policy=compact

# the demo is an ordinary Java process: split lines, groupBy word, count()
bin/kafka-run-class.sh org.apache.kafka.streams.examples.wordcount.WordCountDemo &

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic streams-plaintext-input        # type: all streams lead to kafka

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic streams-wordcount-output --from-beginning \
  --formatter-property print.key=true \
  --formatter-property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
  --formatter-property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer

# the app created its own repartition and changelog topics
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list | grep '^streams-wordcount-'

See it in a real stack

No architecture uses Kafka Streams yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2015–2026

By 2015, teams processing Kafka data had two choices, as Guozhang Wang set out in KIP-28 that July. They could wrap the plain producer and consumer clients in their own loop and handle partitioning, state and failures themselves, or adopt a framework such as Storm, Samza, Spark Streaming or Flink, which ran as its own cluster and used Kafka only at the edges. Jay Kreps, who had helped build Samza at LinkedIn, had concluded that most stream processing looked less like a real-time MapReduce and more like ordinary services, which need the same deployment and monitoring as everything else.

So Kafka Streams became a library. Kreps announced a preview on Confluent’s blog on 10 March 2016, and it shipped in Kafka 0.10.0 on 22 May. “You don’t need to set up any kind of special Kafka Streams cluster and there is no cluster manager, nimbus, daemon processes, or anything like that,” he wrote. Kafka itself did the coordination: consumer groups split the work and topics backed up the state. Kafka 0.11.0 (2017) added transactions, and with them exactly-once processing.

Confluent built KSQL, later ksqlDB, on top of it in 2017. In Kafka 4.2 (February 2026) a new server-side rebalance protocol, KIP-1071, reached general availability with a limited feature set, moving task assignment from the clients to the brokers.

Why it’s built this way

A library instead of a cluster
Kreps wanted stream processors packaged and deployed like any other app, with Docker, Kubernetes, Puppet or a plain command line, and rolling restarts that already worked, rather than through a framework’s own scheduler.
Local state, backed up to Kafka
Lookups in a local store are fast and need no remote database. Writing each change to a changelog topic means a task can be rebuilt on any machine by replaying it.
Kafka partitions as the unit of work
Reusing consumer groups gave assignment and failover for free, and since tasks share no state, threads and instances need no coordination with each other.
Streams and tables as one idea
A table is the latest value per key of a stream, and a stream is the changelog of a table. Modelling both (KStream, KTable) lets one API do joins, aggregations and state backup.

Timeline

  1. 2015

    Guozhang Wang proposes a stream processing client for Kafka in KIP-28

  2. 2016

    Preview announced in March; ships in Kafka 0.10.0 in May

  3. 2017

    Exactly-once processing arrives with Kafka 0.11.0; Confluent builds KSQL on it

  4. 2026

    Kafka 4.2 makes the broker-side Streams rebalance protocol generally available

Sources17 · facts checked 5 October 2026

Improve this page on GitHub