Cloud Parts

Apache ZooKeeper

The coordination service from Yahoo that a generation of Hadoop-era systems use for leader election, membership and configuration.

What it is
Coordination services
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
6 October 2026
Website Source code

What it is

ZooKeeper is a Java server that runs as an ensemble of usually three or five machines. Its data is a tree of small nodes called znodes, named like file paths (/app/config), each holding a little data: kilobytes, with a cap just under 1 MB. The whole tree is kept in memory, backed by a transaction log and snapshots on disk. A client connects to any one server. Reads are answered from that server’s copy, which makes them fast but possibly a little behind, so a client that needs the latest value calls sync first. Writes are forwarded to the leader, which orders them and broadcasts them to the followers with its own protocol, Zab; a write commits once a majority has it. ZooKeeper does best when reads outnumber writes about ten to one.

There is deliberately no lock API. Instead ZooKeeper offers building blocks. Ephemeral znodes are deleted when the session that created them ends, so a crashed client’s entries vanish. Sequential znodes get a rising counter appended to their name. Watches send a client one notification when a znode or its children change, and since version 3.6 a persistent, recursive watch can follow a whole subtree. From these, clients build locks, leader elections, barriers and queues: for a lock, each client creates an ephemeral sequential child, the lowest number holds the lock, and every other client watches only the one just ahead of it. Apache Curator packages these recipes for Java.

HBase requires it, HDFS uses it for automatic NameNode failover, and Solr, Druid, NiFi and Fluss rely on it for coordination, such as electing leaders. Kafka was its best-known user until Kafka 4.0 (March 2025) replaced it with its own Raft quorum. The current release is 3.9.6 and the stable line is 3.8.

Why teams pick it

  • Proven for well over a decade underneath HBase, HDFS, Solr, Druid and, for most of its life, Kafka
  • Any server answers reads from memory, so read-heavy coordination stays fast
  • Ephemeral and sequential znodes make locks, elections and membership lists simple, ordered and free of thundering herds
  • Apache Curator provides tested recipes such as leader latches, locks and service discovery

Watch out for

  • Reads can be stale unless you call sync; only writes are linearizable
  • A standard watch fires once and must be set again, so a client can miss changes that happen in between
  • A JVM service with its own wire protocol (Jute): heap and disk tuning to do, and first-class clients mainly for Java and C
  • Losing ground as systems build Raft in: Kafka removed it in 4.0, Druid 38 moved segment discovery off it, and Fluss plans to drop it

Alternatives

What all coordination services have in common →
etcdOpen sourceThe Raft-replicated key-value store that holds every Kubernetes cluster’s state, built for small, critical data that many machines must agree on.

How it fits with other technologies

Persistent, ephemeral and sequential znodes, and a watch, in the CLI · bash

bin/zkCli.sh -server zk1:2181,zk2:2181,zk3:2181

create /workers
# ephemeral: deleted automatically when this session ends
create -e /workers/worker-7 10.0.3.7

# sequential: ZooKeeper appends a rising counter, the basis of fair locks
create /locks
create -s -e /locks/job- worker-7        # Created /locks/job-0000000003

# be notified once when the list of workers changes
ls -w /workers

See it in a real stack

No architecture uses Apache ZooKeeper yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2008–2025

In the late 2000s Yahoo ran many distributed systems that each needed the same things: a leader, a list of live servers, shared configuration. Its web crawler’s Fetching Service and the Yahoo! Message Broker, a publish-subscribe system with thousands of topics, are two examples. Google had answered the same need with Chubby, a lock service. Patrick Hunt, Mahadev Konar, Flavio Junqueira and Benjamin Reed took a different route: a “coordination kernel” of simple, wait-free data objects with watches and strict ordering, from which applications build whatever primitives they need.

ZooKeeper moved to the Apache Software Foundation in 2008 as part of the Hadoop project, and its authors described it at USENIX ATC in 2010, the year it became a top-level Apache project. It became the default coordination layer for the Hadoop ecosystem and beyond: HBase, HDFS failover, Kafka, Solr and Druid. Apache Curator, which packages the common recipes, graduated to a top-level project in 2013.

Newer coordination stores built on Raft, etcd and Consul, offered alternatives, and some systems began to build consensus in. In 2019 Kafka’s KIP-500 proposed replacing ZooKeeper with a metadata quorum inside Kafka, and Kafka 4.0 shipped without ZooKeeper in March 2025, with thanks to the ZooKeeper community. ZooKeeper itself remains maintained, with 3.9 as its current line.

Why it’s built this way

No locks on the server, only wait-free objects
Blocking primitives let slow or faulty clients hurt fast ones, and make the server depend on other clients’ responses and failures. Exposing simple objects let each application build exactly the primitive it needed.
Reads served locally, writes ordered by one leader
Coordination workloads are mostly reads; Yahoo’s crawler saw between 10 and 100 reads per write. Answering reads from each server’s memory scales them, while Zab gives every write a single global order.
Ephemeral and sequential znodes
Tying a znode to a session turns a client crash into a node disappearing, which is failure detection for free. Sequence numbers give every contender a place in line, so a lock can pass to the next client without waking all of them.
Watches like cache invalidations
A client can cache, say, the current leader and be told once when it changes, instead of polling, the same idea distributed file systems use to keep caches fresh.

Timeline

  1. 2008

    Comes to the Apache Software Foundation, under the Hadoop project

  2. 2010

    USENIX ATC paper; becomes a top-level Apache project

  3. 2013

    Apache Curator, the recipe library, graduates to a top-level project

  4. 2019

    Kafka’s KIP-500 proposes replacing ZooKeeper with a built-in quorum

  5. 2025

    Kafka 4.0 ships without ZooKeeper

Sources21 · facts checked 6 October 2026

Improve this page on GitHub