cloudparts

Apache BookKeeper

A replicated log storage service from Yahoo!: clients append entries to ledgers that are striped and copied across storage servers called bookies, with metadata in ZooKeeper. Apache Pulsar keeps its messages in it.

Apache · Top-level project
What it is
Event streams
Type
Open source
Made by
Apache Software Foundation
Foundation
The Apache Software Foundation, top-level project
License
Apache 2.0
Facts checked
24 October 2026
Website Source code

What it is

The unit of storage is a ledger, an append-only sequence of entries with a single writer. When a client creates a ledger it picks three numbers: the ensemble size E (how many bookies hold the ledger), the write quorum Qw (how many bookies each entry is sent to) and the ack quorum Qa (how many must confirm before the write counts), with E ≥ Qw ≥ Qa. Entries are striped across the ensemble, so each bookie holds fragments of many ledgers rather than whole ones, and the cluster tolerates Qa − 1 failures without losing data. If a bookie fails mid-write, the client swaps in another one and starts a new fragment, recording the change in the metadata store with a compare-and-swap. Each entry also carries the last entry the writer had confirmed, which readers use to follow the log.

A ledger’s metadata, its ensembles and its state (open, in recovery or closed), lives in ZooKeeper. If a writer dies, a reader opens the ledger in recovery mode: it fences the bookies so the old writer can no longer append, works out the last confirmed entry, re-replicates what is needed and closes the ledger, so exactly one writer ever wins. On each bookie, writes go first to a journal on a fast disk for durability and are then batched into entry log files, with an index of where each entry sits; garbage collection and compaction reclaim space from deleted ledgers. Applications build longer logs as ordered lists of ledgers: Pulsar’s managed ledger does this for every topic, and BookKeeper’s own DistributedLog API offers a log stream without handling ledgers directly. Release 4.18.1 came out in September 2026.

Why teams pick it

  • Low-latency, durable writes: a separate journal disk takes the sequential writes while entry logs handle reads
  • Replication and quorum sizes are set per ledger, and striping spreads load across all bookies
  • Fencing gives a clear single-writer guarantee, which makes it a sound base for write-ahead logs and replicated state
  • Storage scales by adding bookies, independently of the brokers or services that write to it

Watch out for

  • A building block, not a message broker: most people use it only underneath Pulsar or another system
  • Another cluster to run, plus ZooKeeper for metadata; disks for journals and entry logs need separate sizing
  • Ledgers can’t be appended to after they close, so applications must manage chains of ledgers and their deletion

Specs

Compare with alternatives →
Data model
Ledgers: append-only sequences of entries, one writer each
Consistency
Single writer per ledger, enforced by fencing on recovery
How it scales
Add bookies; each ledger’s entries are striped across its ensemble
Replication
Per ledger: ensemble size, write quorum and ack quorum
Storage engine
Journal on a fast disk, then entry log files with an index
Ordering
Per ledger, in append order
Retention
Until a ledger is deleted; garbage collection and compaction reclaim space
Ways to run it
Self-hosted, with ZooKeeper for metadata

Alternatives

What all event streams have in common →Compare side by side →
Apache KafkaOpen sourceThe open-source distributed log that most event streaming is built on, or built to be compatible with.+1 Cruise Control for KafkaApache PulsarOpen sourceA multi-tenant messaging and streaming platform from Yahoo that keeps brokers stateless and stores messages in Apache BookKeeper, so one topic can serve as a log, a pub-sub feed or a work queue.PravegaArchivedOpen sourceA stream storage service from Dell that kept events in Apache BookKeeper and moved them to long-term storage, with streams that split and merge their segments automatically as load changes; archived by the CNCF in 2025.

How it fits with other technologies

Create a ledger with ensemble 3, write quorum 3, ack quorum 2, append, and reopen it with fencing · java

WriteHandle wh = bk.newCreateLedgerOp()
    .withDigestType(DigestType.CRC32)
    .withPassword(password)
    .withEnsembleSize(3)
    .withWriteQuorumSize(3)
    .withAckQuorumSize(2)
    .execute()
    .get();

CompletableFuture<Long> addFuture = wh.append("Some entry data".getBytes());

// opening in recovery mode fences the ledger: the old writer can't append any more
ReadHandle rh = bk.newOpenLedgerOp()
    .withLedgerId(ledgerId)
    .withPassword(password)
    .withRecovery(true)
    .execute()
    .get();

See it in a real stack

No architecture uses Apache BookKeeper yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2009–2026

BookKeeper was developed at Yahoo! as a distributed write-ahead log: a way to log streams of records reliably across many servers, keeping its metadata in ZooKeeper, another Yahoo! project. In January 2009 it was contributed to Apache as part of ZooKeeper, described as a system where servers are bookies, logs are ledgers and each record is a ledger entry. In 2011 Benjamin Reed moved the code, together with the Hedwig publish-subscribe system built on it, into its own subproject, and BookKeeper became a top-level Apache project in November 2014.

Yahoo! built its Pulsar messaging service on BookKeeper, using separate disks for the journal and for general storage, and open-sourced Pulsar in September 2016, which made BookKeeper the storage layer of a widely used messaging system. DistributedLog, a replicated log service that entered the Apache Incubator in 2016, became part of BookKeeper in 2017 and now ships as its log stream API. BookKeeper still releases regularly, with 4.17 and 4.18 lines maintained in 2026.

Why it’s built this way

One writer per ledger, enforced by fencing
Logs are only consistent if two writers can never interleave entries. Fencing lets a reader take over a dead writer’s ledger and seal it without asking the writer.
Separate ensemble, write and ack quorums
Applications trade latency against durability differently; striping over a larger ensemble spreads load, and acknowledging after Qa bookies keeps tail latency down when one bookie is slow.
Journal plus interleaved entry logs
Writing every entry to a dedicated journal first gives durable, sequential writes, while entries from many ledgers are batched into shared entry logs so reads don’t compete with the journal.

Timeline

  1. 2009

    BookKeeper contributed to Apache ZooKeeper in January

  2. 2011

    Code moved into its own BookKeeper subproject, with Hedwig

  3. 2014

    Top-level Apache project in November

  4. 2016

    Yahoo! open-sources Pulsar, built on BookKeeper, in September

  5. 2017

    DistributedLog joins BookKeeper from the Apache Incubator

  6. 2026

    4.18.0 in June and 4.18.1 in September

Sources6 · facts checked 24 October 2026

Improve this page on GitHub