Cloud Parts

Table formats

A metadata layer that turns a folder of data files into a real table, with transactions, schema changes and time travel.

12 technologies on this page ↓
Live model
What you’re watchingTwo writers commit to one table. The catalog accepts one commit at a time, so a writer that loses a race gets a conflict and retries. Each accepted commit is a new numbered snapshot. Dashboards follow the latest snapshot, while an audit job reads an older one. Snapshots past retention are expired, and the audit skips ahead.

How it works

Apache Iceberg, Delta Lake and Apache Hudi keep metadata files next to the Parquet data files. Each commit writes new data files plus a new snapshot listing exactly which files make up the table, then makes it current in one atomic step: swapping a pointer in a catalog (Iceberg) or adding the next entry to a transaction log (Delta Lake). That gives you ACID transactions on object storage. Readers always see a complete snapshot, concurrent writers detect conflicts and retry, and old snapshots stay readable for time travel until you expire them. Adding or renaming a column is a metadata change, so it doesn’t rewrite the data.

Think of it as

The catalogue of a library whose books never move. The catalogue says which books make up the collection right now, and you can pull last year’s edition to see what it said then.

Reach for it when

  • Tables in object storage that several engines, such as Spark, Trino, Snowflake and DuckDB, must read and write safely
  • Updates, deletes and upserts on lake data, such as erasing a customer or applying change-data capture
  • Auditing or reproducing a report exactly as of a past snapshot

Think twice when

  • Many small single-row writes per second; each commit adds files and metadata, so batch them or use a database
  • A handful of files that one tool reads; plain Parquet is enough

A real example

A streaming job commits new orders to an Iceberg table every minute while a nightly job compacts small files and deletes rows for customers who asked to be erased. Analysts in Trino and data scientists in Spark query the same table and never see a half-written commit. Finance reruns last quarter’s report against the snapshot from 1 July.

More on table formats

Narrower ideas that build on this one.

Technologies

Specific products, and what each does differently.

Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.PuffinOpen sourceApache Iceberg’s file format for the data that doesn’t fit in a manifest: a container of typed binary blobs, such as distinct-value sketches and deletion vectors, described by a JSON footer.Mainly File formatsApache PolarisOpen sourceAn open-source Iceberg REST catalog: one server that every engine asks where a table’s current metadata is, with role-based access control and short-lived storage credentials per table.Mainly Lakehouse catalogsAmazon S3 TablesAWSA bucket type for Apache Iceberg tables: S3 stores them, acts as their catalog, and compacts and cleans them up for you.Mainly LakehousesApache AmoroIncubatingOpen sourceAn incubating Apache “lakehouse management system”, started at NetEase as Arctic, that watches Iceberg tables and keeps them fast on its own: it compacts small files, rewrites deletes and expires old snapshots and files in the background.Mainly Lakehouses

What each provider calls it

AWS
S3 Tables (managed Iceberg), Glue Data Catalog, Lake Formation
Google Cloud
Apache Iceberg managed tables in BigQuery, Lakehouse runtime catalog (formerly BigLake metastore)
Azure
Microsoft Fabric OneLake (Delta Lake), Azure Databricks (Unity Catalog)
Open source & others
Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon, Lance (for AI data); catalogs such as Apache Polaris, Unity Catalog and Nessie

See it in a real stack

Architectures that use table formats

Small lakehouse on DuckLakeDuckDB writes and queries tables whose catalog lives in PostgreSQL and whose data sits as Parquet on S3.DuckDBDuckLakePostgreSQLAmazon S3Open lakehouse on IcebergKafka streams events into Spark, which writes Iceberg tables on S3 that Trino queries for dashboards.Apache KafkaApache SparkApache IcebergAmazon S3Trino

In code · sql

-- Spark SQL with an Iceberg catalog named "lake"
CREATE TABLE lake.sales.orders (
  order_id BIGINT, customer_id BIGINT, amount DECIMAL(10,2), ts TIMESTAMP
) USING iceberg
PARTITIONED BY (days(ts));

MERGE INTO lake.sales.orders t
USING updates u ON t.order_id = u.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

-- time travel: the table as it was on 1 July
SELECT sum(amount) FROM lake.sales.orders TIMESTAMP AS OF '2026-07-01 00:00:00';

Often confused with

File formatsLakehousesLakehouse catalogsQuery enginesRelational databasesFile format or table format?
Where it came from2009–2025

Hive, built at Facebook and described in a 2009 VLDB paper, gave Hadoop its idea of a table: an HDFS directory of files, split into partition subdirectories such as ds=20090101, with a metastore recording each table’s schema and location. That worked for batch loads, but the files were the truth. Readers found data by listing directories, writers changed tables by adding and renaming files, and nothing stopped a query from seeing half of an update.

Cloud object stores made this worse. Listing millions of objects was slow, consistency guarantees were limited, and a rename on S3 is really a copy. Between 2016 and 2017 three companies built a fix independently: Uber built Hudi to apply database changes to its lake within minutes, Netflix started Iceberg to replace Hive’s layout with a spec that tracks every file, and Databricks launched Delta Lake after cloud storage problems caused around half of its support escalations. All three record the exact list of files in each table version and switch versions atomically.

All three were open source by 2019 and competed for years before converging. Delta’s UniForm (2023) writes Iceberg and Hudi metadata, Apache XTable converts between formats, and in 2024 Databricks agreed to acquire Tabular, the company founded by Iceberg’s creators. Newer designs such as DuckLake (2025) move the metadata into a SQL database altogether.

Why it’s built this way

Track files explicitly, not directories
Listing directories is slow on object stores and can’t tell a finished write from a half-written one. A manifest or log that names every file makes the table’s contents exact.
One atomic step per commit
Object stores have no multi-object transactions, but a single small write can be made atomic: swapping a pointer in a catalog (Iceberg) or creating the next log file (Delta Lake). Writers that lose the race retry.
Immutable data files plus delete markers
Parquet files can’t be edited in place, so updates write new files, and Iceberg v2 records deleted rows in separate delete files instead of rewriting whole files.
Keep old snapshots
Because files are never modified, an old version costs only storage. That gives time travel and reproducible reads until the snapshot is expired.

Timeline

  1. 2009

    Hive paper describes tables as directories plus a metastore

  2. 2016

    Uber builds Hudi to apply updates to its Hadoop lake

  3. 2017

    Iceberg starts at Netflix; Databricks launches Delta for customers

  4. 2019

    Delta Lake open-sourced; Hudi joins Iceberg in the Apache Incubator

  5. 2023

    Delta Lake UniForm writes Iceberg and Hudi metadata

  6. 2024

    XTable enters the Apache Incubator; Databricks agrees to buy Tabular

  7. 2025

    DuckLake keeps table metadata in SQL; Iceberg adds v3 deletion vectors

Sources21

Improve this page on GitHub