Cloud Parts

Message queues

A to-do list between services: each job is done once, by one worker.

11 technologies on this page ↓
Live model
What you’re watchingCheckout bursts faster than two workers can keep up, so jobs pile up in the queue. Each worker takes the next job only when it’s free, and every job is handled once.

How it works

Producers drop messages into the queue and move on. Workers pull a message, process it and delete it when done. If a worker crashes mid-job, the message reappears after a visibility timeout and another worker retries it. The queue absorbs bursts so a slow system downstream isn’t overwhelmed, and you go faster by adding workers.

Think of it as

The ticket rail in a restaurant kitchen. Waiters clip orders up, cooks take the next one, and each order is cooked exactly once.

Reach for it when

  • Background jobs: emails, reports, image processing
  • Smoothing bursts so a database or API sees a steady rate
  • Retrying flaky work, with a dead-letter queue for messages that keep failing

Think twice when

  • Several systems each need every message; use pub/sub or a stream
  • You may want to replay old messages; once deleted they’re gone

A real example

A checkout service enqueues “send receipt” jobs. During a flash sale 5,000 arrive in one minute. Two workers drain them over the next few minutes, and the email provider never sees more than its rate limit.

Technologies

Specific products, and what each does differently.

Amazon SQSAWSAWS’s fully managed message queue: no brokers to run, billed per request.BullMQOpen sourceThe Node.js job queue on Redis, Bull’s successor, with ports to other languages and, since v6, a Postgres backend.Cloudflare QueuesCommercialA message queue built into Cloudflare Workers: Workers send messages through a binding, and Queues delivers them in batches to a consumer Worker or to any HTTP client that pulls them, at least once.pgmqOpen sourceAn SQS-style message queue made of SQL functions and tables inside Postgres.PgQueOpen sourceSkype’s PgQ engine repackaged as one SQL file: a zero-bloat event log inside any Postgres 14+.PgQueuerOpen sourceA Python background-job library that uses Postgres as the broker: `SKIP LOCKED` to claim, LISTEN/NOTIFY to wake.RabbitMQOpen sourceA widely used open-source message broker that routes messages to queues through exchanges.

Also used for message queues

Products of other kinds that are often used this way, grouped by what they are.

What each provider calls it

AWS
SQS
Google Cloud
Cloud Tasks, Pub/Sub pull subscriptions
Azure
Service Bus queues, Storage queues
Open source & others
RabbitMQ, Redis + BullMQ, Celery; on Postgres: pgmq, PgQueuer, PgQue

See it in a real stack

Architectures that use message queues

CQRS with a static read sideCommands update PostgreSQL; a Lambda projector rewrites the affected JSON views in S3, and CloudFront serves every read without touching a database.Amazon API GatewayAWS LambdaPostgreSQLAmazon SQSAmazon S3Amazon CloudFrontBackground jobs with a queueThe API drops slow work onto SQS and returns at once; a pool of Fargate workers drains the queue at a steady pace.AWS LambdaAmazon SQSAWS FargateAmazon S3

In code · python

import boto3
sqs = boto3.client("sqs")
URL = "https://sqs.us-east-1.amazonaws.com/123456789012/receipts"

sqs.send_message(QueueUrl=URL, MessageBody='{"order_id": 4812}')

# worker loop: receive, process, then delete
resp = sqs.receive_message(QueueUrl=URL, WaitTimeSeconds=20)
for msg in resp.get("Messages", []):
    send_receipt(msg["Body"])
    sqs.delete_message(QueueUrl=URL, ReceiptHandle=msg["ReceiptHandle"])

Worth exploring

Article· June 2026Durable Queue Workers With Just PostgresFatih Yildiz (mfyz)Starts with the FOR UPDATE SKIP LOCKED queue, then shows what to add when a job has several steps: Absurd stores each step’s result in Postgres, so a crashed worker resumes from the last checkpoint. The author calls it a fit for small and medium projects, not high volume.
Article· June 2026Making Postgres Queues ScaleQian Li and Peter Kraft, DBOSHow DBOS got its Postgres-backed queues to 30,000 workflows a second: SKIP LOCKED, the weaker READ COMMITTED isolation unless a queue needs global limits, partial indexes on waiting rows only, and an index that matches the dequeue order. DBOS wrote it about its own product, but the techniques apply to any Postgres queue.
Article· April 2026Keeping a Postgres queue healthySimeon Griggs, PlanetScaleWhy a busy queue table fills with dead rows, and how one long query elsewhere can stop vacuum from cleaning them up and slow every dequeue. The diagnosis applies to any Postgres; the fix it ends on, Traffic Control, exists only on PlanetScale.
Video· April 2022Queues in PostgreSQLThomas Munro, PostgreSQL committer, at Citus Con 2022A Postgres developer walks through what queue-like workloads do inside the database, and the complications of using Postgres as a job or message queue.
All resources →

Often confused with

Event streamsPub/subServerless functions (FaaS)Workflow orchestrationQueue, stream or pub/sub?
Where it came from1965–2026

Message queues are almost as old as networked computing. IBM announced its Queued Telecommunications Access Method (QTAM) for System/360 mainframes in 1965, and banks and airlines pushed the idea further, because a lost, corrupted or duplicated transaction meant lost money or a lost seat. The point, as IBM’s Andy Stanford-Clark puts it, was to let systems work without being in “strict synchronous lockstep”: the sender drops a message and carries on, and the receiver picks it up when it’s ready.

IBM’s MQSeries, introduced in 1993, made the queue a product that connected applications across different machines, and, according to IBM, is still used by 98 of the top 100 global banks. Microsoft followed with MSMQ in 1997, and Java’s JMS API (1998) let Java programs use existing products such as MQSeries. Those products were proprietary and didn’t talk to each other, so in 2003 John O’Hara at JPMorgan began designing the open AMQP protocol, which RabbitMQ implemented as open source in 2007.

Cloud providers then turned the queue into a utility. Amazon SQS, in beta from 2004 and generally available in 2006, had no broker to run and charged per request; Google, Microsoft and others followed. The idea keeps returning: Kafka, built as a log, made queue-style consumption production-ready in 2026.

Why it’s built this way

Store and forward instead of direct calls
The queue holds the message, so sender and receiver don’t need to be up, or equally fast, at the same time. Bursts pile up in the queue instead of overwhelming the receiver.
Each message goes to one consumer
Competing workers share the load, so throughput grows by adding workers, and each job is done once rather than by everyone.
Delete only after acknowledgement
A message is removed only when a consumer confirms it, so a crash mid-job means redelivery rather than loss. The trade-off is that a consumer can see a message twice and must be idempotent.
No history after delivery
Queues were built to move work, not store it, which keeps them simple. Replaying old messages needs a stream instead.

Timeline

  1. 1965

    IBM announces QTAM for System/360 mainframes

  2. 1993

    IBM introduces MQSeries

  3. 1998

    JMS gives Java a standard API for messaging products

  4. 2003

    JPMorgan begins designing the open AMQP protocol

  5. 2006

    Amazon SQS becomes generally available after a 2004 beta

  6. 2007

    RabbitMQ, an open-source AMQP broker, is released

  7. 2026

    Kafka 4.2 makes queue-style share groups production-ready

Sources14

Improve this page on GitHub