cloudparts

StreambedPreview

A small Go program that copies Postgres changes into Iceberg or DuckLake tables on S3 through logical replication, and answers SQL over them through a Postgres-compatible server built on DuckDB.

What it is
ETL & ELT pipelines
Used for
Lakehouses,Change data capture
Type
Open source
License
Apache 2.0
Status
Preview. No tagged release yet: the changelog lists 0.1.0 as upcoming, and almost all commits come from one developer since April 2026.
Facts checked
23 October 2026
Website Source code

What it is

Streambed connects to Postgres as a logical replication subscriber, so the source needs wal_level = logical and nothing else. One loop reads the WAL through the pgoutput plugin, decodes inserts, updates and deletes, and buffers rows per table in memory. Every 10,000 rows or 2 seconds by default it writes the buffer as a Parquet file to S3 (or MinIO or another S3-compatible store) and commits a new table snapshot. Updates and deletes are copy-on-write by default: Streambed reads the affected Parquet data, drops the old rows and writes the merged result; Iceberg v2 equality deletes are an opt-in alternative. It only acknowledges a WAL position to Postgres once the rows before it are on S3, so Postgres keeps any WAL it hasn’t flushed, and resync backfills a table with COPY under a consistent snapshot.

Tables can be written as Apache Iceberg, readable by any Iceberg engine, or as DuckLake, whose catalog lives in a DuckDB or SQLite file. The built-in query server embeds DuckDB and speaks the Postgres wire protocol, so you can point psql at port 5433 and run analytical queries, including time travel to a past commit with AT (TIMESTAMP => …). Other commands list snapshots, expire old Iceberg snapshots and compact small files. With Postgres 16 or newer it can read changes from a hot-standby replica, using the primary only to create the publication.

Why teams pick it

  • Very little to run: one Go binary, a Postgres with logical replication, and a bucket; no Kafka, Spark or warehouse in between
  • Data lands in open formats in your own bucket, as Parquet with Iceberg or DuckLake metadata, so other engines can read the same tables
  • Analysts query with the Postgres tools they know, through a DuckDB-backed server that answers columnar queries much faster than the row-store source
  • Can stream from a read replica to keep load off the primary, and keeps a snapshot per commit for time travel

Watch out for

  • Very young: started in April 2026, no tagged release, and essentially one maintainer
  • Copy-on-write rewrites Parquet data for every batch of updates and deletes, and frequent small commits pile up Iceberg metadata and small files; the author’s own benchmark found DuckLake about 38 times faster than Iceberg on an update-heavy load (7 s against 269 s)
  • Time-travel timestamps are Streambed’s commit times, not the original Postgres transaction times, and expired snapshots can’t be queried
  • Equality-delete (merge-on-read) mode is one-way: once a table has equality deletes, Streambed refuses to start in copy-on-write mode

Alternatives

What all etl & elt pipelines have in common →Compare side by side →
MeltanoOpen sourceA command-line ELT tool that runs Singer taps and targets from a meltano.yml project kept in git.OLakeOpen sourceAn open-source tool in Go that copies databases, Kafka topics and S3 files into Apache Iceberg tables (or plain Parquet), with full loads, incremental syncs and CDC, and no Kafka, Debezium, Spark or Flink in between.Apache NiFiOpen sourceA Java dataflow server you program by drawing a graph of processors in a web UI, with buffered queues, back pressure and a record of where every piece of data went.Apache CamelOpen sourceA Java integration framework that moves and transforms messages between systems with routes built from Enterprise Integration Patterns, written in Java, XML or YAML, with over 350 connectors.DebeziumOpen sourceOpen-source change data capture: connectors that read a database’s transaction log and emit every committed insert, update and delete as an event, usually into Kafka.

How it fits with other technologies

Stream a local Postgres into Iceberg on MinIO and query it with psql · bash

./streambed sync \
  --source-url="postgres://postgres:test@localhost:5432/postgres" \
  --s3-bucket="streambed" \
  --s3-endpoint="http://localhost:9000" \
  --s3-prefix="test" \
  --query-addr=:5433

# analytical queries go to DuckDB over Parquet, not to your primary
psql -h localhost -p 5433 -U postgres -d postgres

See it in a real stack

No architecture uses Streambed yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2026

Analytical queries on a production Postgres database eventually become too slow to run there but too important to drop. The usual answer is an ETL pipeline into a warehouse, with Spark, Airflow or a managed connector service, hours of delay and a copy that can drift from the source. Vignesh (viggy28 on GitHub), formerly at Cloudflare and a co-founder of Omnigres, started Streambed in April 2026 to skip that pipeline: subscribe to the WAL directly and write open table files to S3, seconds behind production.

The first version wrote only Iceberg. In August 2026 he added DuckLake as a second target after finding that change-data capture, with its stream of small commits, multiplies Iceberg metadata files and small Parquet files; DuckLake keeps that metadata in a database instead and fit the write path, which already used DuckDB. Reading from a hot-standby replica followed in September.

Why it’s built this way

Logical replication straight into table files
Skipping Kafka, batch jobs and Spark removes the moving parts and the hours of delay of a classic pipeline, and keeps the data seconds behind the source.
A Postgres wire protocol query server on DuckDB
Users keep psql and their Postgres tools, while DuckDB’s columnar engine on Parquet handles the analytical queries that are slow on Postgres’s row storage.
One goroutine for the whole pipeline
Reading, decoding, buffering, flushing and acknowledging in one loop, with no channels between stages, keeps ordering and backpressure easy to reason about.
DuckLake alongside Iceberg
Small, frequent CDC commits create a lot of Iceberg metadata in object storage; a catalog in a database gives a shorter commit path, while Iceberg remains for engine interoperability.

Timeline

  1. 2026

    Repository created in April; “Why I built Streambed” published on 12 April

  2. 2026

    DuckLake target added in August

  3. 2026

    Streaming from a hot-standby replica in September

Sources8 · facts checked 23 October 2026

Improve this page on GitHub