PgQueuer

A Python background-job library that uses Postgres as the broker: SKIP LOCKED to claim, LISTEN/NOTIFY to wake.

What it is: Message queues
Type: Open source
Made by: janbjorge (JeyBee) and contributors
License: MIT
Facts checked: 6 October 2026

What it is

PgQueuer is a Python library that turns a Postgres database into a job queue. pgq install creates a pgqueuer table, a log table and a trigger. A producer calls queries.enqueue(entrypoint, payload), which inserts a row, and the trigger sends a NOTIFY that wakes listening workers. A worker, started with pgq run module:factory, claims up to batch_size ready rows with FOR UPDATE SKIP LOCKED, marks them picked and runs the async handler registered for that entrypoint. While a job runs, the worker refreshes its heartbeat; if the heartbeat is older than heartbeat_timeout (30 seconds by default), another worker can pick the job up again. On success the row is deleted from the queue and the outcome appended to pgqueuer_log. Workers also poll, so a missed notification only delays a job.

Because jobs are rows, a job can be enqueued in the same transaction as the data it’s about. PgQueuer adds cron-style schedules, deferred jobs (execute_after), retries with a delay, per-entrypoint concurrency limits enforced across all workers, deduplication keys, Prometheus metrics, a terminal dashboard and an in-memory mode for tests. It’s free under the MIT licence, needs Python 3.10+ and Postgres 14+, and reached 1.0 in May 2026 after two years of 0.x releases.

Why teams pick it

Transactional enqueue: the job commits or rolls back with your data, so there’s no dual write to drift
Jobs start right away through LISTEN/NOTIFY, with polling as a safety net
The usual job-queue features built in: cron schedules, deferred jobs, retries with backoff, cancellation, completion tracking and fleet-wide concurrency limits
Observability out of the box: Prometheus metrics, tracing with Logfire or Sentry, and pgq dashboard
Tests and prototypes can run without Postgres using PgQueuer.in_memory()

Watch out for

Python only: handlers must be async def, and retries, cancellation and tracing are Python APIs, so workers in other languages aren’t an option
At least once: if a worker dies mid-job the job runs again, and a crash re-delivery doesn’t increase attempts, so handlers must be idempotent
Each finished job is a delete on a high-churn table; without the aggressive autovacuum settings from pgq autovac, dead tuples slow index scans
Claims, heartbeats and logs load the same Postgres instance as your application, and pgqueuer_log is never pruned automatically
No workflows: no chains, groups or DAGs, and nothing stores a handler’s return value

Alternatives

What all message queues have in common →

Amazon SQSAWSAWS’s fully managed message queue: no brokers to run, billed per request.

BullMQOpen sourceThe Node.js job queue on Redis, Bull’s successor, with ports to other languages and, since v6, a Postgres backend.

Cloudflare QueuesCommercialA message queue built into Cloudflare Workers: Workers send messages through a binding, and Queues delivers them in batches to a consumer Worker or to any HTTP client that pulls them, at least once.pgmqOpen sourceAn SQS-style message queue made of SQL functions and tables inside Postgres.

PgQueOpen sourceSkype’s PgQ engine repackaged as one SQL file: a zero-bloat event log inside any Postgres 14+.

RabbitMQOpen sourceA widely used open-source message broker that routes messages to queues through exchanges.

How it fits with other technologies

What PgQueuer uses

PgQueuerkeeps jobs, locks and its log inPostgreSQLRelational databases

A worker with a concurrency limit, and a transactional enqueue · python

# worker.py: run with `pgq run worker:main`
import asyncpg
from pgqueuer import PgQueuer
from pgqueuer.db import AsyncpgDriver
from pgqueuer.models import Job
from pgqueuer.queries import Queries

async def main() -> PgQueuer:
    conn = await asyncpg.connect()  # uses PGHOST, PGUSER, PGDATABASE
    pgq = PgQueuer(AsyncpgDriver(conn))

    @pgq.entrypoint("send_receipt", concurrency_limit=4)
    async def send_receipt(job: Job) -> None:
        order_id = int(job.payload.decode())
        print(f"emailing receipt for order {order_id}")

    return pgq

# in the web app: the job commits only if the order does
async def place_order(conn: asyncpg.Connection, order_id: int) -> None:
    queries = Queries(AsyncpgDriver(conn))
    async with conn.transaction():
        await conn.execute("INSERT INTO orders (id, status) VALUES ($1, 'paid')", order_id)
        await queries.enqueue("send_receipt", str(order_id).encode())

pgmq, PgQue and PgQueuer#

All three keep queues in Postgres tables. They differ in what a reader gets, which languages can join in, and how old rows are cleaned up.

	pgmq	PgQue	PgQueuer
Model	Message queue: each message goes to one reader	Shared event log: every subscribed consumer sees every event	Job queue: Python workers run a handler per job
What you install	SQL objects, as an extension or a plain SQL file	One PL/pgSQL file, plus a scheduler that calls the ticker	A Python package; `pgq install` creates its tables
Languages	Any, through SQL; official Rust and Python clients	Any, through SQL; first-party Python, Go and TypeScript clients	Python 3.10+, async handlers
Handing out work	`FOR UPDATE SKIP LOCKED`, then a visibility timeout	Batches between ticks, found by comparing two transaction snapshots; no row locks	`FOR UPDATE SKIP LOCKED`, then status `picked` with a heartbeat
Delivery	Exactly once within the visibility timeout, then at least once	At least once, in batches; about 52 ms median wait at the default 100 ms tick	At least once; handlers must be idempotent
Retries	The message reappears after the timeout; track `read_ct` yourself	`nack` schedules a retry; a dead-letter queue after the limit	`RetryRequested` with a delay, or hold failed jobs for review
Finished messages	Deleted, or moved to an archive table	Stay in the log; `ack` moves the consumer’s cursor	Deleted from the queue; the outcome is appended to a log table
Dead rows and cleanup	Updates and deletes leave dead rows for VACUUM; optional partitions dropped by pg_partman	None on the hot path: three tables rotate and the oldest is emptied with `TRUNCATE`	Deletes leave dead rows; `pgq autovac` makes autovacuum aggressive
Licence and maturity	PostgreSQL licence; 1.0 in November 2023	Apache 2.0; 0.x, first release April 2026	MIT; 1.0 in May 2026

PgQue’s latency figure comes from its own committed benchmark.

See it in a real stack

No architecture uses PgQueuer yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2024–2026

Celery, a long-standing Python job queue, runs beside a separate broker such as Redis, RabbitMQ or SQS, plus a celery-beat process for recurring tasks. Yet Postgres already has what a queue needs: row locks with SKIP LOCKED, so workers don’t collide, and LISTEN/NOTIFY to wake them. In April 2024 a developer who publishes on GitHub as janbjorge (JeyBee) released PgQueuer to use just those, so an application that already has Postgres needs no other broker.

Early Show HN posts in April and May 2024 advertised over 5,000 jobs a second, and a post that August drew 376 points. Features arrived in quick 0.x releases, such as recurring schedules in 0.15 (November 2024) and deduplication keys in 0.20 (April 2025).

After about two years and 26 minor releases, 1.0 shipped on 11 May 2026 as the first stable release, with strict semantic versioning from then on. The project records its design in architecture decision records: workers claim jobs by lock contention, notifications only signal and never carry job data, and delivery is at least once.

Why it’s built this way

Workers race for rows; no coordinator assigns work: With FOR UPDATE SKIP LOCKED, any worker can claim any ready job and the database arbitrates, so there’s no coordinator or worker registry to run. The cost is no fairness between workers and lock traffic at the head of the queue.
NOTIFY wakes workers but never carries the job: NOTIFY payloads are capped at about 8 kB, vanish if the sending transaction rolls back, and are missed by late listeners. So a wake-up only prompts a normal claim query, and a periodic poll covers anything missed.
At-least-once, not exactly-once: Exactly-once would need fencing tokens or an outbox and still breaks once side effects leave the database. Re-running a crashed job was judged better than losing it.

Timeline

2024
0.1 released in April; early posts claim over 5,000 jobs a second
2024
0.15 adds recurring, cron-style schedules
2025
0.20 adds job deduplication with a dedupe_key
2026
1.0, the first stable release, ships in May
2026
1.4 makes concurrency limits hold across simultaneous workers

Sources21 · facts checked 6 October 2026

Improve this page on GitHub