Cloud Parts

Coordination services

A small, strongly consistent store, usually three or five servers, that a distributed system uses to agree on who leads, who holds a lock and what the configuration is.

4 technologies on this page ↓
Live model
What you’re watchingA five-member cluster. Services send every write to the leader, which copies it to all four followers; a write is committed once a majority (three of five, counting the leader) has it. Two followers can go down and writes still succeed. When the leader itself fails, writes stop until the others elect a new one, about a second with etcd’s defaults (the model reuses the same box for the new leader).

How it works

When many machines share a job, they keep needing to agree on small facts: which copy is the leader, who holds a lock, which servers are alive, what the current settings are. A coordination service is a dedicated cluster, usually three or five servers, that answers those questions so that every client sees the same answer. The servers run a consensus protocol (Paxos in Google’s Chubby, Zab in ZooKeeper, Raft in etcd and Consul). One server is elected leader and every write goes through it. The leader adds the write to a replicated log and counts it as committed only once a majority, called a quorum, has stored it. Any two majorities share at least one server, so a committed write is never lost, and a minority cut off by a network split stops accepting writes instead of disagreeing. That is why clusters have an odd number of members: five servers survive two failures, and a sixth adds nothing. On top of the log sit a few primitives. Watches tell a client when a key changes, so it doesn’t poll. Leases and sessions tie a key to a client’s heartbeats, so an ephemeral key, such as a lock or an “I’m alive” record, disappears when its owner dies. Compare-and-set and sequential keys let many clients race for a lock and exactly one win. Every write is replicated through one leader and synced to disk, so these systems hold kilobytes to a few gigabytes of metadata, never your application’s data: etcd defaults to a 2 GiB quota and suggests 8 GiB at most, and ZooKeeper keeps everything in memory with each node capped just under 1 MB.

Think of it as

A club committee keeping the official minute book. A decision counts only once most members have signed it, so losing one or two members changes nothing, and two halves of a split committee can never both pass rules. The committee doesn’t store the club’s belongings, only who holds which key.

Reach for it when

  • Electing one active leader among several copies of a service: a database primary, a scheduler, a controller
  • Locks and leases for work that must happen in exactly one place at a time
  • Small shared configuration and membership lists that many machines watch for changes
  • Metadata for a bigger system, like Kubernetes objects in etcd or Vitess’s shard map

Think twice when

  • Application data, large values or heavy write traffic: a few GB at most, and every write goes through one leader
  • Caches and counters that can live with stale reads; a key-value store or cache is faster and scales out
  • Lock-protected writes without a fencing token: a paused client can outlive its lease, so pass the lock’s revision (or ZooKeeper’s zxid) to the storage and have it reject older ones

A real example

Three copies of a billing scheduler must never charge customers twice in one night. Each starts by trying to create the key /billing/leader in etcd, attached to a 10-second lease it keeps renewing. The one whose transaction succeeds runs the jobs; the other two watch the key. If the leader’s machine dies, its renewals stop, the lease expires within 10 seconds, the key disappears, and the watchers race again until one wins. The five-member etcd cluster underneath keeps working through the loss of any two members.

Technologies

Specific products, and what each does differently.

Also used for coordination services

Products of other kinds that are often used this way, grouped by what they are.

What each provider calls it

AWS
No general-purpose coordination service. Amazon EKS runs etcd (three instances across three Availability Zones) inside each cluster’s control plane; Amazon MSK runs ZooKeeper for ZooKeeper-mode Kafka clusters; AWS Cloud Map for service discovery
Google Cloud
No standalone service. GKE keeps each cluster’s state in etcd on the control plane or in Spanner, managed for you; Service Directory for service discovery
Azure
No standalone service. AKS manages etcd in its control plane, and HDInsight uses ZooKeeper to elect the active head node
Open source & others
etcd (Apache 2.0, CNCF), Apache ZooKeeper with the Apache Curator client library, Consul (Business Source License since 1.17); Google’s Chubby is internal only

In code · bash

# every write gets a new cluster-wide revision number
etcdctl put /config/feature-x on
etcdctl get /config/feature-x -w json        # shows mod_revision

# be told about changes instead of polling
etcdctl watch /config/ --prefix

# a key that vanishes when its owner stops renewing the lease
etcdctl lease grant 10                        # lease 694d7bc81a3c5e2f granted with TTL(10s)
etcdctl put /members/worker-7 10.0.3.7 --lease=694d7bc81a3c5e2f
etcdctl lease keep-alive 694d7bc81a3c5e2f

# hold a named lock while a command runs
etcdctl lock nightly-billing ./charge-customers.sh

Often confused with

Key-value storesService discovery & meshContainer orchestrationRelational databases
Where it came from1990–2025

In the late 1980s Leslie Lamport set out to prove that Echo, a fault-tolerant file system built at the SRC research lab, was trying to do the impossible: stay consistent through failures and keep working while any majority of its machines was up. Instead he found an algorithm that does exactly that, and described it as the parliament of an imaginary Greek island, Paxos. The paper, first submitted in 1990, appeared only in 1998. For years consensus stayed mostly academic, and each distributed system improvised its own failure handling.

In 2006 Google’s Mike Burrows described Chubby. Rather than ask every team to build Paxos into its programs, Google ran a small lock service, typically five replicas per cell, that also stored small files. GFS used a Chubby lock to appoint its master; Bigtable used Chubby to elect a master, find its servers and let clients find the master. Engineers at Yahoo built ZooKeeper for the same needs but chose a lower-level API: a tree of small data nodes with watches, “Chubby without the lock methods”, from which clients build their own locks, queues and elections. It came to Apache in 2008, was described in a 2010 paper, and became the coordination layer of the Hadoop generation, from HBase to Kafka.

In 2013 CoreOS announced etcd, a key-value store for shared configuration and service discovery, built on Raft, a consensus algorithm Diego Ongaro and John Ousterhout designed at Stanford to be easier to understand than Paxos and published in 2014. HashiCorp’s Consul, released in 2014, used Raft too. Kubernetes made etcd the store for all of its cluster state. Since then, some systems have absorbed consensus instead of depending on a separate service: Kafka 4.0 (2025) replaced ZooKeeper with its own Raft quorum.

Why it’s built this way

A separate service instead of a consensus library in every program
Chubby’s designers found that systems usually start as prototypes and need high availability only later. Taking a lock to become master took a couple of statements; restructuring an existing server to take part in a consensus protocol was far harder.
Majority quorums and odd cluster sizes
Any two majorities overlap, so a committed write survives failures and two halves of a split cluster can never both accept writes. Four members tolerate no more failures than three, so clusters run three, five or at most seven members.
Watches and invalidations instead of polling or timeouts
Chubby pushed invalidations to clients that cached data, and its developers liked not having to pick a cache timeout like DNS’s time-to-live. ZooKeeper’s watches let a client cache, say, the current leader and hear the moment it changes.
Keys tied to a client’s heartbeats
If a lock holder crashes, someone must notice and free the lock. Chubby sessions, ZooKeeper’s ephemeral nodes and etcd’s leases all delete a client’s keys when its keep-alives stop, which turns failure detection into a key disappearing.

Timeline

  1. 1990

    Lamport submits “The Part-Time Parliament”, describing Paxos; it is published in 1998

  2. 2006

    Google describes Chubby at OSDI

  3. 2008

    ZooKeeper, built at Yahoo, comes to the Apache Software Foundation

  4. 2010

    ZooKeeper paper at USENIX ATC; it becomes a top-level Apache project

  5. 2013

    CoreOS announces etcd, built on Raft

  6. 2014

    Raft paper at USENIX ATC; HashiCorp releases Consul

  7. 2020

    etcd graduates in the CNCF

  8. 2025

    Kafka 4.0 runs without ZooKeeper, on its own Raft quorum

Sources23

Improve this page on GitHub