Background jobs with retries
Handing work to a pool of workers, each job done once, retried on failure, and set aside when it keeps failing.
What it asks of a system
A job queue holds units of work (send this email, resize this image, charge this card) until a worker takes one. The job stays hidden from other workers while it is being processed and is deleted only when the worker says it is done; if the worker crashes or times out, the job reappears and another worker tries again. After a set number of failed attempts it moves to a dead-letter queue, so one bad job doesn’t loop forever and someone can look at it later.
To tell whether a system really handles it, check what it offers for retries (attempt limits, backoff, a dead-letter queue), for delayed and scheduled jobs, for priorities and per-key ordering, and for long-running jobs that need their lease extended. Almost every queue delivers at least once, so a job can run twice; workers have to be idempotent, or the queue has to offer deduplication.
Good at it · 8
Compare side by sideMessage queues
Amazon SQSAWSReceived messages hide for a visibility timeout and reappear if not deleted, and a redrive policy moves repeat failures to a dead-letter queue.AWS’s fully managed message queue: no brokers to run, billed per request.
BullMQOpen sourceJobs get attempts with fixed or exponential backoff, delays, priorities, rate limits and cron schedules, and stalled jobs go back to waiting.The Node.js job queue on Redis, Bull’s successor, with ports to other languages and, since v6, a Postgres backend.
Cloudflare QueuesCommercialBatches go to a consumer Worker that scales out with the backlog, with per-message ack and retry, delays and a dead-letter queue.A message queue built into Cloudflare Workers: Workers send messages through a binding, and Queues delivers them in batches to a consumer Worker or to any HTTP client that pulls them, at least once.pgmqOpen sourceAn SQS-style visibility timeout in Postgres: unfinished messages reappear, and a message can be sent in the same transaction as its data.An SQS-style message queue made of SQL functions and tables inside Postgres.
PgQueuerOpen sourcePostgres rows claimed with SKIP LOCKED, plus heartbeats, retries with delay, schedules and per-entrypoint concurrency limits.A Python background-job library that uses Postgres as the broker: SKIP LOCKED to claim, LISTEN/NOTIFY to wake.
RabbitMQOpen sourceThe broker pushes jobs up to a prefetch limit, removes each on ack, and sends rejected or expired ones to a dead-letter exchange.A widely used open-source message broker that routes messages to queues through exchanges.
Sources
- AWS docs: Amazon SQS visibility timeout
- AWS docs: Amazon SQS dead-letter queues
- BullMQ docs: Retrying failing jobs (attempts, fixed and exponential backoff)
Why each technology is listed comes from the sources on its own page.