Cloud Parts

DuckLake

A lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.

What it is
Table formats
Used for
Lakehouses
Type
Open source
Made by
The DuckDB team
License
MIT
Facts checked
5 October 2026
Website Source code

What it is

Iceberg and Delta Lake keep a table’s metadata as files next to the data: manifests, snapshots and logs that every reader has to fetch and stitch together. DuckLake moves all of that into a SQL database instead. The catalog (which files belong to which table, every snapshot, the schema history) lives in PostgreSQL, SQLite, MySQL or a DuckDB file. The data itself stays as plain Parquet files on S3, another object store, or local disk.

A commit is one database transaction, so changes across several tables can be atomic, and finding the files for a query is one SQL lookup instead of a chain of object-store reads. Small writes can be held in the catalog database first (“data inlining”) and flushed to Parquet later, which avoids the small-files problem. DuckDB has first-class support through its ducklake extension, and DuckLake 1.0 shipped in April 2026 with a backward-compatibility guarantee.

Why teams pick it

  • No catalog server to run: any database you already operate, such as PostgreSQL, is the catalog
  • Snapshots, time travel, schema evolution and multi-table ACID transactions
  • Data is ordinary Parquet, so it isn’t locked in; you can copy tables out to Iceberg
  • Fast small commits, because metadata updates are database rows rather than new files

Watch out for

  • DuckDB’s ducklake extension is the reference implementation; other engines, such as DataFusion, are only starting to add support
  • The catalog database sets the limit: if it’s slow or down, so is every table in the lake
  • A DuckDB-file catalog allows only one client; use PostgreSQL for many readers and writers
  • Younger than Iceberg and Delta Lake, with a smaller ecosystem of tools

Alternatives

What all table formats have in common →
Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.

How it fits with other technologies

A shared DuckLake with a PostgreSQL catalog and data on S3 · sql

INSTALL ducklake;

ATTACH 'ducklake:postgres:dbname=lake host=catalog.internal' AS lake
  (DATA_PATH 's3://acme-lake/data/');
USE lake;

CREATE TABLE orders AS FROM 's3://acme-raw/orders/*.csv';
INSERT INTO orders FROM 's3://acme-raw/orders-today.csv';

-- read the table as it was before today’s insert
SELECT count(*) FROM orders AT (VERSION => 1);

See it in a real stack

Small lakehouse on DuckLake →
Live model
What you’re watchingA nightly load writes new rows: DuckLake commits the snapshot to PostgreSQL and the data to S3 as Parquet. Analysts’ queries look up the current files in the catalog and read them from S3.
Where it came from2025–2026

Iceberg and Delta Lake made tables on object storage practical, but they keep their metadata as files. Iceberg uses what DuckLake’s authors called “a maze of JSON and Avro files” to record schemas, snapshots and which Parquet files belong to a table. Every small change writes more metadata files, and reading a table means a chain of sequential HTTP requests before the first byte of data. To find the current version safely, both formats also ended up adding a catalog service, which is itself backed by a database.

On 27 May 2025, Mark Raasveldt and Hannes Mühleisen of the DuckDB project proposed DuckLake: if a database is needed anyway, put all the metadata in it and keep only the data in Parquet. They pointed out that Google BigQuery (with Spanner) and Snowflake (with FoundationDB) already manage metadata this way, “just without the open formats at the bottom.” The spec stores catalog, snapshots and statistics as ordinary tables in PostgreSQL, MySQL, SQLite or DuckDB.

Version 0.3 added Iceberg interoperability in September 2025, and DuckLake 1.0 shipped on 13 April 2026 with a stable spec and a backward-compatibility guarantee. Engines beyond DuckDB have started to follow, such as an Apache DataFusion integration in July 2026.

Why it’s built this way

Metadata in a SQL database, data in Parquet
The authors’ view is that blob storage is great for scaling data files, but tracking snapshots, schemas and statistics is a job for a database, as BigQuery and Snowflake already do internally.
A commit is one database transaction
Iceberg and Delta Lake essentially manage one table at a time. Using the database’s own transactions gives atomic changes across many tables for free.
Small changes can be inlined into the catalog
Small writes are the lake’s worst case: each one adds files to compact later. DuckLake can keep small inserts as rows in the metadata database, which allows sub-millisecond writes.
Any ACID SQL database can be the catalog
There is no catalog server to deploy: a DuckDB file works on a laptop, and PostgreSQL works for many readers and writers.

Timeline

  1. 2025

    DuckLake announced on 27 May with a spec and a DuckDB extension

  2. 2025

    DuckLake 0.3 adds Iceberg interoperability and geometry support (September)

  3. 2026

    DuckLake 1.0 ships in April with a backward-compatibility guarantee

  4. 2026

    An Apache DataFusion integration is announced in July

Sources11 · facts checked 5 October 2026

Improve this page on GitHub