Messaging
How services hand work and events to each other.
7 concepts · 43 technologies · 2 comparisons · 5 real stacks
Concepts
Start with Message queues- Message queues11 technologiesA to-do list between services: each job is done once, by one worker.
- Event streams10 technologiesAn append-only log of events that many readers replay at their own pace.
- Stream processing16 technologiesQueries that never finish: they read events as they arrive and keep results, counts and joins up to date.
- Pub/sub10 technologiesPublish one message; every subscriber gets its own copy.
- Event buses2 technologiesA router that sends each event to whoever has a rule for it.
- Workflow orchestration6 technologiesA durable state machine that drives multi-step processes to completion.
- IoT device messaging6 technologiesA broker that keeps a connection open to every device in a fleet, knows who each one is, and copes with devices that keep dropping offline.
Technologies
Grouped by the concept each one mainly implementsCommonly confused
See it in a real stack
Architectures that use messaging conceptsOpen lakehouse on IcebergKafka streams events into Spark, which writes Iceberg tables on S3 that Trino queries for dashboards.Apache KafkaApache SparkApache IcebergAmazon S3TrinoEvent-driven order processingCheckout publishes one event to Kafka; payments, inventory and email each consume it at their own pace.Apache KafkaPostgreSQLAWS LambdaCQRS with a static read sideCommands update PostgreSQL; a Lambda projector rewrites the affected JSON views in S3, and CloudFront serves every read without touching a database.Amazon API GatewayAWS LambdaPostgreSQLAmazon SQSAmazon S3Amazon CloudFrontBackground jobs with a queueThe API drops slow work onto SQS and returns at once; a pool of Fargate workers drains the queue at a steady pace.AWS LambdaAmazon SQSAWS FargateAmazon S3IoT telemetry pipelineDevices stream readings into Kinesis; Lambda writes them to TimescaleDB for Grafana, and raw data is archived to S3.Amazon Kinesis Data StreamsAWS LambdaTimescaleDBAmazon S3Grafana
Worth exploring
All resources →Article· June 2026Durable Queue Workers With Just PostgresFatih Yildiz (mfyz)Starts with the
FOR UPDATE SKIP LOCKED queue, then shows what to add when a job has several steps: Absurd stores each step’s result in Postgres, so a crashed worker resumes from the last checkpoint. The author calls it a fit for small and medium projects, not high volume.Article· June 2026Making Postgres Queues ScaleQian Li and Peter Kraft, DBOSHow DBOS got its Postgres-backed queues to 30,000 workflows a second:
SKIP LOCKED, the weaker READ COMMITTED isolation unless a queue needs global limits, partial indexes on waiting rows only, and an index that matches the dequeue order. DBOS wrote it about its own product, but the techniques apply to any Postgres queue.Article· April 2026Keeping a Postgres queue healthySimeon Griggs, PlanetScaleWhy a busy queue table fills with dead rows, and how one long query elsewhere can stop vacuum from cleaning them up and slow every dequeue. The diagnosis applies to any Postgres; the fix it ends on, Traffic Control, exists only on PlanetScale.
Slides· September 2025How and Why Datadog moved from cloud-managed to self-managed PostgresNick Canzoneri, Datadog, at PGConf NYC 2025Slides from a talk on why a company running hundreds of Postgres clusters across several clouds left managed services for its own Postgres platform on Kubernetes, how it moved a whole region, and what got better and worse. The stack is Patroni (with ZooKeeper for leader election), WAL-G backups, a Helm chart rather than an operator, and Temporal workflows for automation. Useful before you weigh RDS-style convenience against running Postgres yourself.
Article· September 2025Understanding Apache FlussJack VanlightlyA long, detailed walk through Fluss internals, reverse-engineered from the code: log tables, primary-key tables that emit changelogs, and tiering to lakehouse tables. The author works at Confluent, which runs Flink but not Fluss, and says so up front.
Article· August 2025lambda to eks pod migrationAWS re:Post community questionA short Q&A on what replaces Lambda’s built-in triggers, SQS polling and scaling when functions move to EKS pods. The useful answer points to KEDA (with Karpenter) for scaling pods on queue depth; the longer answer is an unreviewed AI-generated checklist from AWS’s own assistant, so treat it as a starting list, not guidance.
Article· October 2024Stream Processing With Apache Flink: book notesChris Shumaker, Maker ConsultingOne engineer’s chapter-by-chapter notes on the O’Reilly book by Fabian Hueske and Vasiliki Kalavri, read to understand what owning a Flink job in production involves: state, time, scale and performance. A quick way to decide whether the book is worth reading in full.
Video· April 2022Queues in PostgreSQLThomas Munro, PostgreSQL committer, at Citus Con 2022A Postgres developer walks through what queue-like workloads do inside the database, and the complications of using Postgres as a job or message queue.
Article· May 2021Spice up Your Kubernetes Environment with AWS LambdaLiav Yona, MediumShows how a Lambda function triggered by SQS or SNS can forward events to an internal (ClusterIP) service in EKS through the Kubernetes API server’s service proxy, using IAM-to-RBAC mapping instead of exposing an endpoint, with code on GitHub. A clever pattern, but written for 2021-era EKS: it maps the IAM role through the
aws-auth ConfigMap, and it routes application traffic through the cluster’s control plane.


















