cloudparts

DuckLake or Iceberg?

DimensionDuckLakeApache Iceberg
In one lineTable metadata as rows in a SQL database, data as Parquet filesTable metadata as a tree of JSON and Avro files next to the data, found through a catalog
Where metadata livesOrdinary tables in a catalog database: PostgreSQL for many users, SQLite or a DuckDB file locally; MySQL works but isn’t recommendedFiles in object storage: a metadata JSON file, manifest lists and Avro manifests; the catalog stores only the location of the current metadata file
Commit pathOne database transaction. Snapshot ids are a primary key, so of two clashing writers one fails, then retries without rewriting data files unless the changes really conflictWrite data files and new metadata files, then swap the catalog’s pointer atomically; if another commit got there first, retry against the new snapshot
Changes across tablesAtomic, through the database’s own transactionsAtomic only through a REST catalog’s multi-table commit endpoint, where the catalog implements it
Small writesInlined: inserts and deletes of up to 10 rows by default are stored in the catalog database and flushed to Parquet later, for example by CHECKPOINTEvery commit writes at least one data file plus new metadata files, so frequent small commits pile up files until compaction runs
Deletes and updatesA positional delete file in Parquet per affected data file; deletion vectors in Puffin files are experimental in 1.0Copy-on-write rewrites, or merge-on-read with position or equality delete files (v2) and deletion vectors in Puffin files (v3)
CatalogThe catalog database is the catalog; there is no separate service to runA separate catalog to choose and run: a REST catalog such as Polaris or Lakekeeper, AWS Glue, Nessie or a JDBC database
EnginesDuckDB’s ducklake extension is the reference; DataFusion, Spark (from MotherDuck) and two Trino implementations come from othersSpark, Flink, Trino, DuckDB, ClickHouse, Snowflake, BigQuery, Athena, Redshift and many more
MaintenanceMerge adjacent files, expire snapshots, delete old files and rewrite files with many deletes; CHECKPOINT runs them together. Vacuum the catalog database now and thenCompact data files, rewrite delete files, expire snapshots and remove orphan files, on a schedule or through a managed service
What limits scaleThe catalog database’s performance, according to the FAQMetadata reads and commit conflicts: many snapshots and manifests slow planning, and many writers to one table retry
InteroperabilityCopies tables to and from Iceberg since 0.3; bucket partitioning uses Iceberg’s murmur3 hashingXTable can translate its metadata to and from Delta Lake and Hudi
MaturityAnnounced in May 2025; 1.0 in April 2026 with a backward-compatibility guaranteeStarted at Netflix in 2017, Apache top-level project since 2020; spec versions 1 to 3 complete, 4 in development
Licence and governanceMIT, for both the spec and the DuckDB extension; developed by the DuckDB teamApache 2.0; an Apache Software Foundation project

Rule of thumb

Pick DuckLake when DuckDB is your main engine and you already run PostgreSQL: there is no catalog service, small writes are cheap and multi-table transactions come free. Pick Iceberg when many engines and cloud services must read and write the same tables. The choice isn’t final: both keep data as Parquet, and DuckLake can copy tables out to Iceberg.

Sources