Cloud Parts

Apache Iceberg

The most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.

What it is
Table formats
Used for
Lakehouses
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
5 October 2026
Website Source code

What it is

An Iceberg table is a tree of files. At the top, a table metadata file records the schema, the partition spec and the list of snapshots. Each snapshot points to a manifest list, which points to manifests, which list the data files (usually Parquet) with column statistics for each file. A catalog, such as a REST catalog, AWS Glue or a JDBC database, stores where the current metadata file is, and a commit swaps that pointer atomically.

Because there are statistics at every level, an engine can rule out whole manifests without listing any directories. Partitioning is hidden: you partition by days(ts), queries just filter on ts, and you can change the scheme later without rewriting old data. Netflix built Iceberg and contributed it to the Apache Software Foundation in 2018. Spec versions 1, 2 and 3 are complete; v3 adds deletion vectors, row lineage, column default values and new types such as variant, geometry and nanosecond timestamps. Version 4 is still in development.

Why teams pick it

  • Wide engine support: Spark, Flink, Trino, DuckDB, ClickHouse, Snowflake, BigQuery, Athena and Redshift all work with it
  • The REST catalog protocol is an open standard, so one catalog can serve every engine
  • Hidden partitioning and partition evolution avoid accidental full scans and painful re-partitioning
  • Managed offerings on the big clouds, such as Amazon S3 Tables and Google’s Apache Iceberg managed tables in BigQuery

Watch out for

  • You need a catalog, and choosing one (Glue, Polaris, Nessie, a JDBC database and others) is a real decision
  • Frequent small commits pile up snapshots, manifests and small files, so compaction and snapshot expiry must run on a schedule
  • Engines adopt new spec versions unevenly: Athena works only with v2 tables, and Trino’s v3 support is experimental
  • Many concurrent writers to one table cause commit conflicts and retries

Alternatives

What all table formats have in common →
Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.

How it fits with other technologies

What uses Apache Iceberg

Table maintenance in Spark SQL: evolve partitioning, list snapshots, compact, expire · sql

-- "lake" is an Iceberg catalog; the table was partitioned by days(ts)
ALTER TABLE lake.sales.orders REPLACE PARTITION FIELD ts_day WITH hours(ts);

SELECT snapshot_id, committed_at, operation
FROM lake.sales.orders.snapshots
ORDER BY committed_at DESC;

CALL lake.system.rewrite_data_files(table => 'sales.orders');
CALL lake.system.expire_snapshots(
  table => 'sales.orders',
  older_than => TIMESTAMP '2026-09-01 00:00:00');

See it in a real stack

Open lakehouse on Iceberg →
Live model
What you’re watchingServices publish order events to Kafka. Spark reads them in micro-batches and commits them to an Iceberg table whose files live on S3. Dashboard queries run on Trino, which reads the same table.
Where it came from2017–2025

Netflix ran its analytics on Hive tables, the de facto standard layout that Hive, Presto and Spark all used. A Hive table tracks its partitions in a central metastore and finds the files inside by listing directories. On S3 that listing is slow, and while S3 listings were eventually consistent it could return wrong results. Changes to a table couldn’t be atomic, object stores have no real rename, and the metastore became a bottleneck when planning jobs.

Ryan Blue and Dan Weeks started Iceberg at Netflix in 2017 as “a new table format for storing large, slow-moving tabular data”, one that tracks every data file instead of directories. Netflix donated it to the Apache Software Foundation, where it entered the incubator in November 2018 and graduated in May 2020. Blue, Weeks and Jason Reid later founded Tabular, a data management company built on Iceberg.

Iceberg became the format most engines agreed on. In June 2024 Databricks, the company behind rival Delta Lake, agreed to acquire Tabular with the aim of making the two formats compatible, and in December 2024 AWS launched S3 Tables, which builds Iceberg support into S3 itself.

Why it’s built this way

Track individual files, not directories
Listing partition paths was slow on S3 and gave no atomicity. Iceberg’s metadata names every data file, so writers create files where they like and commit them explicitly, and planning a scan needs no listing.
Commit by swapping one pointer
Each change writes new metadata, then atomically replaces the pointer to the current metadata file. Readers never take locks and never see a partial write: the spec’s goal is serializable isolation.
A written spec, independent of any engine
Hive’s layout was a convention built into each engine. Iceberg wrote the table format down as a specification, so Spark, Trino, Flink and others could each implement it and read the same tables.
Schema and partition changes are metadata-only
The spec requires safe add, drop, rename and reorder of columns, including nested ones, and partition spec evolution, so changing a table’s design never means rewriting old data.

Timeline

  1. 2017

    Ryan Blue and Dan Weeks start Iceberg at Netflix

  2. 2018

    Donated to the Apache Software Foundation; enters the incubator in November

  3. 2020

    Graduates to a top-level Apache project in May

  4. 2021

    Format v2 tables, with row-level deletes, arrive in Iceberg 0.12

  5. 2022

    Iceberg 1.0 released in November

  6. 2024

    Databricks agrees to buy Tabular (June); AWS launches S3 Tables (December)

  7. 2025

    Iceberg 1.8 adds format v3 features: deletion vectors, variant, row lineage

Sources18 · facts checked 5 October 2026

Improve this page on GitHub