Cloud Parts

Apache Pulsar

A multi-tenant messaging and streaming platform from Yahoo that keeps brokers stateless and stores messages in Apache BookKeeper, so one topic can serve as a log, a pub-sub feed or a work queue.

What it is
Event streams
Used for
Pub/sub,Message queues
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
8 October 2026
Website Source code

What it is

Pulsar splits serving from storage. Brokers are stateless: each topic is owned by one broker, which takes writes from producers, dispatches messages to consumers and serves most reads from a cache. The messages themselves live in Apache BookKeeper, a separate cluster of storage nodes called bookies. A topic is stored as a managed ledger made of many BookKeeper ledgers, append-only logs with a single writer whose entries are copied to several bookies; consumer positions (cursors) are stored there too. Because brokers hold no data, a topic can move to another broker in seconds, and storage grows by adding bookies. A metadata store keeps cluster state: Oxia (recommended for new clusters) or Apache ZooKeeper, and RocksDB in standalone mode. Old data can be offloaded with tiered storage to Amazon S3, Google Cloud Storage, Azure Blob Storage or a file system, and clients still read it as if nothing changed.

Topics are named persistent://tenant/namespace/topic. Tenants and namespaces are built in, with their own authentication, quotas, retention and isolation policies, so one cluster can serve many teams. How a topic behaves depends on the subscription: exclusive (one consumer, the default), failover (one active consumer with standbys), shared (round-robin across consumers, each message acknowledged individually, no ordering) and key_shared (spread across consumers but each key always goes to the same one). A unique subscription per consumer gives pub-sub fan-out; a shared subscription turns the same topic into a queue, with delayed delivery, negative acks, retry and dead-letter topics. Geo-replication copies topic data between clusters in different regions. Pulsar Functions run small Java, Python or Go functions on each message inside the cluster, and Pulsar IO connectors move data to and from systems such as Kafka and Cassandra. Pulsar 5.0 (October 2026), the newest long-term-support release, adds scalable topics that split and merge key ranges as load changes, instead of a fixed partition count.

Why teams pick it

  • Brokers and storage scale separately: stateless brokers take over topics in seconds, and adding bookies adds storage capacity right away
  • One system for streaming and queuing, chosen per subscription (exclusive, failover, shared, key_shared), with delayed delivery and dead-letter topics built in
  • Multi-tenancy and geo-replication are part of the core, from Yahoo’s need to run messaging as a hosted service across 10+ data centres
  • Tiered storage offloads old backlog to S3, GCS or Azure Blob, so long retention doesn’t need large bookie disks

Watch out for

  • More to run than a single-binary broker: brokers, a BookKeeper cluster and a metadata store, plus a configuration store for multi-cluster setups
  • Each topic is served by one broker, so a hot topic needs partitions (or 5.0’s scalable topics) to spread load
  • Shared subscriptions give up ordering, and key_shared needs producers to disable batching or batch by key
  • No Kafka protocol in the core, and the Kafka-on-Pulsar protocol handler is archived (last updated January 2024), so moving data between Kafka and Pulsar goes through Pulsar IO connectors

Alternatives

What all event streams have in common →
Amazon Kinesis Data StreamsAWSAWS’s serverless event stream, split into shards that each keep records in order.Apache FlussOpen sourceStreaming storage built for Flink: event logs and updatable tables in columnar Arrow format, with key lookups, and older data copied into Paimon, Iceberg, Hudi or Lance tables.Apache KafkaOpen sourceThe open-source distributed log that most event streaming is built on, or built to be compatible with.Cloudflare K2BetaCommercialCloudflare’s serverless event stream: a durable log stored in R2 that Workers and HTTP clients append to, read through independent subscriptions that lease out batches of records.RedpandaCommercialA Kafka-compatible streaming broker rewritten in C++ as one binary, with no JVM and no ZooKeeper.WarpStreamCommercialA Kafka-compatible stream that keeps no data on disks: stateless Agents write straight to your object storage bucket.

How it fits with other technologies

A tenant, a namespace, and one topic read as a log and as a keyed queue · bash

bin/pulsar standalone &          # broker, bookie and metadata store in one JVM

bin/pulsar-admin tenants create acme
bin/pulsar-admin namespaces create acme/orders
bin/pulsar-admin topics create-partitioned-topic \
  persistent://acme/orders/created --partitions 4

# an exclusive subscription: one consumer reads everything, in order
bin/pulsar-client consume persistent://acme/orders/created \
  -s audit -t Exclusive -p Earliest -n 0 &

# a key_shared subscription: run it twice and the two consumers split
# the keys between them, each key always going to the same consumer
bin/pulsar-client consume persistent://acme/orders/created \
  -s billing -t Key_Shared -n 0 &

# key_shared needs batching off (or key-based batching) on the producer
bin/pulsar-client produce persistent://acme/orders/created \
  -k customer-42 -db -m 'order 1,order 2,order 3'

See it in a real stack

No architecture uses Apache Pulsar yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2015–2026

Yahoo needed one messaging service for many of its products, run centrally with applications onboarded as tenants. It wanted publish latencies around 5 ms on average and under 15 ms at the 99th percentile, durability with every message on several disks, replication between data centres, and support for a million topics. Finding no open-source system that met all of that, its team built Pulsar and deployed the first instance in the second quarter of 2015.

Pulsar was built on Apache BookKeeper, a distributed write-ahead log that Yahoo had open-sourced in 2011, with ZooKeeper for coordination. By September 2016, when Yahoo open-sourced Pulsar under Apache 2.0, it ran in more than 10 data centres with full-mesh replication and over 1.4 million topics, behind Yahoo Mail, Finance, Sports and Gemini Ads. The project entered the Apache Incubator on 1 June 2017 and graduated to a top-level project in September 2018, by which time it had added Pulsar Functions and the Pulsar IO connector framework.

The project now numbers releases as LTS.feature.patch: 4.0 (October 2024) and 5.0 (October 2026) are long-term-support lines, and 5.0 has active support until October 2028.

Why it’s built this way

Separate stateless brokers from BookKeeper storage
Brokers that hold no data can hand topics to each other within seconds, and BookKeeper’s separate journal and storage disks keep consumers draining a backlog from slowing down publishers. The ASF called this two-layer design a key strength at graduation.
Multi-tenant from the start
Yahoo ran Pulsar as a hosted service for many teams, so tenants, namespaces, quotas and access control are core concepts rather than conventions.
Subscription modes instead of separate products
An exclusive subscription behaves like a JMS topic and a shared one like a JMS queue, so the same stored topic serves both patterns.
Replication between data centres in the core
Yahoo had users in many regions who created global topics; brokers asynchronously push messages published locally to remote clusters.

Timeline

  1. 2015

    First Pulsar instance in production at Yahoo, second quarter

  2. 2016

    Open-sourced by Yahoo under Apache 2.0 in September

  3. 2017

    Enters the Apache Incubator on 1 June

  4. 2018

    Graduates to an Apache top-level project in September

  5. 2024

    Pulsar 4.0, a long-term-support release, in October

  6. 2026

    Pulsar 5.0 LTS adds scalable topics, 5 October

Sources18 · facts checked 8 October 2026

Improve this page on GitHub