Cloud Parts

Pub/sub

Publish one message; every subscriber gets its own copy.

10 technologies on this page ↓
Live model
What you’re watchingEach message is copied to all three subscribers. While inventory is offline its copies are lost, which is why durable designs put a queue behind each subscriber.

How it works

A publisher sends a message to a topic without knowing who’s listening. The broker delivers a copy to every current subscriber. Adding a consumer means adding a subscription, with no change to the publisher. Plain pub/sub keeps no history, so an offline subscriber misses messages unless its subscription is backed by a durable queue.

Think of it as

A group chat. You post once and everyone in the group sees it, but people who join later don’t see what was said before.

Reach for it when

  • Fan-out: one event triggers several independent reactions
  • Decoupling teams, since publishers don’t need to know about consumers
  • Device messaging over MQTT, where each device subscribes to its own topics

Think twice when

  • Work that one worker must do exactly once; that’s a queue
  • Consumers that need to replay history; that’s a stream

A real example

When an order is placed, one OrderPlaced message fans out to email, analytics and inventory. A year later the fraud team subscribes too, and the order service doesn’t change at all.

More on pub/sub

Narrower ideas that build on this one.

Technologies

Specific products, and what each does differently.

Also used for pub/sub

Products of other kinds that are often used this way, grouped by what they are.

Event streams →

Apache KafkaOpen sourceThe open-source distributed log that most event streaming is built on, or built to be compatible with.Apache PulsarOpen sourceA multi-tenant messaging and streaming platform from Yahoo that keeps brokers stateless and stores messages in Apache BookKeeper, so one topic can serve as a log, a pub-sub feed or a work queue.Cloudflare K2BetaCommercialCloudflare’s serverless event stream: a durable log stored in R2 that Workers and HTTP clients append to, read through independent subscriptions that lease out batches of records.RedpandaCommercialA Kafka-compatible streaming broker rewritten in C++ as one binary, with no JVM and no ZooKeeper.WarpStreamCommercialA Kafka-compatible stream that keeps no data on disks: stateless Agents write straight to your object storage bucket.

What each provider calls it

AWS
SNS, IoT Core (MQTT)
Google Cloud
Pub/Sub
Azure
Service Bus topics, Web PubSub, Event Grid MQTT broker
Open source & others
MQTT (Mosquitto, EMQX, HiveMQ), Zenoh, NATS, Redis Pub/Sub

In code · python

import boto3, json
sns = boto3.client("sns")

sns.publish(
    TopicArn="arn:aws:sns:us-east-1:123456789012:orders",
    Message=json.dumps({"type": "OrderPlaced", "order_id": 4812}),
)
# Email, analytics and inventory each subscribe to the topic,
# usually through their own SQS queue so nothing is lost.

Often confused with

Message queuesEvent streamsEvent busesIoT device messagingQueue, stream or pub/sub?
Where it came from1985–2019

Publish/subscribe grew up in the mid-1980s, on trading floors and in research labs. Trading rooms needed stock quotes and news on many screens the moment they changed. Vivek Ranadivé’s Teknekron Software Systems, founded in 1985, built an “Information Bus” that delivered that information as it became available, and its successor, TIBCO, carried the idea into enterprises. At Cornell, Ken Birman’s Isis toolkit, released in 1987, included a topic-based “news” service written by Frank Schmuck, which Birman calls “the first widely used publish-subscribe solution.”

In 1999 Andy Stanford-Clark of IBM and Arlen Nipper wrote MQTT to monitor oil pipelines over expensive satellite links, taking pub/sub to small devices on poor networks. Cloud providers later offered topics as a service: Amazon SNS in 2010, and Google Cloud Pub/Sub in 2015, built on infrastructure Google’s own products use for over 500 million messages per second.

The lines have since blurred. Durable subscriptions turn a topic into a set of queues, logs such as Kafka add replay, and MQTT, an OASIS standard since 2014, is widely used for device telemetry.

Why it’s built this way

Publishers address topics, not receivers
A sender doesn’t know who is listening, so new consumers can be added years later without changing the publisher.
Many topics mapped onto few channels
One group per topic created huge numbers of groups, so Isis mapped topics onto a few groups and filtered at the receiver, an approach Birman notes is still common in pub/sub products.
Deliver to whoever is subscribed now
Classic pub/sub keeps nothing, which keeps latency low. Systems that can’t lose messages put a durable queue behind each subscription.
Tiny messages for costly links (MQTT)
MQTT’s control messages can be as small as two bytes, because its first users paid for satellite bandwidth from remote pipelines.

Timeline

  1. 1985

    Teknekron Software Systems founded; builds the Information Bus for trading rooms

  2. 1987

    Cornell’s Isis toolkit released, with a topic-based “news” service

  3. 1999

    MQTT written to monitor oil pipelines over satellite

  4. 2010

    Amazon SNS brings managed topics to the cloud

  5. 2014

    OASIS publishes MQTT 3.1.1 as a standard

  6. 2015

    Google Cloud Pub/Sub becomes generally available

  7. 2019

    MQTT 5 released

Sources12

Improve this page on GitHub