Small lakehouse on DuckLake
DuckDB writes and queries tables whose catalog lives in PostgreSQL and whose data sits as Parquet on S3.
Click any box to open its page. Open it in the playground to rewire it, change the traffic or take parts down.
A lakehouse doesn’t need a fleet of servers. Here a nightly job and a handful of analysts all use DuckDB, an engine that runs inside their own process, and share one set of tables through DuckLake. The catalog that says which Parquet files make up each table, and every past snapshot, is just a few tables in a PostgreSQL database. The data is plain Parquet in an S3 bucket. There’s nothing else to run.
How it flows
- 1
Load runs inside DuckDB
The nightly job is a script that starts DuckDB, runs
ATTACH 'ducklake:postgres:…', and inserts the day’s CSV exports. No cluster starts up; the work happens in the job’s own process.Nightly loadDuckDB - 2
Data lands as Parquet
DuckLake writes the new rows as Parquet files under the bucket’s data path. Existing files are never changed, so a half-finished load can’t corrupt what readers see.
- 3
One transaction commits the snapshot
The list of new files, their statistics and a new snapshot id go into the catalog tables in PostgreSQL in a single transaction. The moment it commits, every reader sees the new data; until then, nobody does.
- 4
Queries ask the catalog, then read files
An analyst’s DuckDB asks PostgreSQL which files belong to the current snapshot and uses their min/max statistics to skip files that can’t match. It then reads only the Parquet it needs from S3.
AT (VERSION => n)reads an older snapshot instead.
Why this shape works
- Almost nothing to operate: one PostgreSQL database and a bucket, both of which you may already run
- Commits are database transactions, so loads that touch several tables are atomic
- Readers scale out for free: each analyst’s laptop or notebook is its own engine
- The data is ordinary Parquet, so other tools can read the files, and tables can be copied out to Iceberg later
Where it strains
- Every query and commit touches PostgreSQL; size and back it up like the critical system it now is
- Each DuckDB runs on one machine, so a single query is limited to what that machine can scan
- Few engines besides DuckDB read DuckLake today; if Spark or Trino must join in, Iceberg is the safer choice
- Many tiny loads still make many small files; use data inlining or run compaction regularly
Common swaps
Avoid egress fees or stay on your own hardware by pointing the data path at another S3-compatible store.
Need many different engines on the same tables? Build the same shape on Iceberg with a REST catalog instead.
Built from
DuckDBOpen sourceQuery enginesAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.
DuckLakeOpen sourceTable formatsA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.
PostgreSQLOpen sourceRelational databasesThe open-source relational database that most new projects start on, extensible enough to grow into many niches.
Amazon S3AWSObject storageAWS’s object storage, and the de facto API most object stores copy.