cloudparts

Cruise Control for Kafka

A service that watches a Kafka cluster’s load, works out which partitions to move to balance it, and moves them: on request through a REST API, or by itself when a broker fails.

Linux Foundation project
What it is
Event streams
Part of
Tool for Apache Kafka
Type
Open source
Made by
Cruise Control for Kafka project (LF Projects, started at LinkedIn)
Foundation
The Linux Foundation, joined July 2026. A series of LF Projects, LLC, started at LinkedIn
License
Apache 2.0
Facts checked
28 October 2026
Website Source code

What it is

Kafka never moves replicas between brokers on its own: when a broker dies, is added or runs hot, someone has to decide which replicas go where and run the reassignment. Cruise Control does that. A metrics reporter plugged into each broker samples CPU, disk and network use per partition and writes it to a Kafka topic (__CruiseControlMetrics); Cruise Control reads that into a load model of the whole cluster. Its analyzer then searches for a set of replica and leader moves that satisfies a priority list of goals, such as rack awareness, staying under capacity limits, and balanced disk, network, CPU, replica and leader counts, and you can plug in goals of your own. The executor applies the moves in batches through Kafka’s own partition reassignment.

Everything goes through a REST API: GET /kafkacruisecontrol/load and proposals show the model and what it would do, and POST rebalance, add_broker, remove_broker and demote_broker act on it. Every POST is a dry run unless you pass dryrun=false. An anomaly detector watches for goal violations, broker and disk failures, metric anomalies and slow brokers, and with self-healing enabled it fixes them without anyone calling the API. Strimzi and Red Hat Streams for Apache Kafka deploy it next to the brokers and drive it from a KafkaRebalance resource. Version 3.1.0, the first release under the Linux Foundation, came in October 2026; it supports Kafka 4.3, needs Java 17, and changed the licence from BSD 2-Clause to Apache 2.0 and the Maven coordinates to io.cruise-control.

Why teams pick it

  • Rebalancing based on measured load, not just replica counts, with many goals weighed together and custom goals possible
  • Self-healing: dead brokers and disks, and goal violations, can be fixed automatically
  • Safe by default: every action is a dry run until you say otherwise, and moves run in limited batches
  • Proven at LinkedIn’s scale and built into Strimzi and other Kafka distributions, now under neutral governance with a steering committee of more than a dozen maintainers

Watch out for

  • Another JVM service to run, with its own configuration, capacity file and a metrics reporter JAR on every broker
  • Moving partitions copies data between brokers, so a large rebalance takes hours and adds load while it runs
  • The load model needs a warm-up of metric samples before proposals are trustworthy
  • Version numbers are confusing: 3.1.0 follows the 2.5.x line, while the earlier 3.0.x releases were LinkedIn’s internal fork

Preview a rebalance, then run it · bash

CC=http://cruise-control:9090/kafkacruisecontrol

# current load per broker, from the metrics reporter samples
curl -s "$CC/load?json=true"

# what a rebalance would move (POSTs are dry runs by default)
curl -s -X POST "$CC/rebalance?json=true"

# do it, then drain broker 4 before decommissioning it
curl -s -X POST "$CC/rebalance?dryrun=false"
curl -s -X POST "$CC/remove_broker?brokerid=4&dryrun=false"

# progress of the running execution
curl -s "$CC/state?substates=executor"

See it in a real stack

No architecture uses Cruise Control for Kafka yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2017–2026

By 2017 LinkedIn ran Kafka on more than 1,800 servers handling over 2 trillion messages a day. Brokers failed daily, and each failure or expansion left workloads unbalanced until site reliability engineers moved partitions by hand. Cruise Control began as an intern project by Efe Gencer and grew into a general service built outside Kafka itself, following a monitor, analyse and act loop. LinkedIn open-sourced it in August 2017, announced by lead engineer Jiangjie (Becket) Qin at Kafka Summit San Francisco.

It became the standard rebalancer for self-managed Kafka, built into Strimzi, Red Hat’s and Cloudera’s distributions. LinkedIn kept a separate 3.0.x line for its internal builds, while the public 2.5.x line served everyone else. In July 2026 the project adopted a technical charter as a series of LF Projects, LLC under the Linux Foundation, with a steering committee chaired by Viktor Somogyi-Vass, and moved to its own GitHub organisation; its first release there, 3.1.0, relicensed it to Apache 2.0.

Why it’s built this way

Live outside the Kafka broker
A separate service could be extended and generalised without changes to Kafka itself, and could be adopted on clusters as they were.
Goals in priority order
Hard goals such as rack awareness and capacity must hold; softer balance goals are optimised only as far as they don’t break the ones above them.
Dry run by default
Moving partitions is expensive and hard to undo, so the API makes you ask for the real thing explicitly.

Timeline

  1. 2017

    LinkedIn open-sources Cruise Control (August)

  2. 2026

    Becomes an LF Projects series (charter adopted 27 July); 3.1.0 under Apache 2.0 (October)

Sources9 · facts checked 28 October 2026

Improve this page on GitHub