Cloud Parts

Lakehouse catalogs

The service every engine asks “where is this table right now?”: it maps table names to their current metadata, makes commits atomic, and decides who may read which files.

7 technologies on this page ↓

How it works

An Iceberg table is a pile of immutable files plus a chain of metadata files, and only one of those metadata files is current. The catalog stores that pointer for each name, such as sales.orders → s3://lake/…/00042-….metadata.json, grouped into namespaces (also called databases or schemas). To commit, a writer writes new data and a new metadata file, then asks the catalog to swap the pointer from the version it started on to its new one. The swap is a compare-and-swap: if another writer got there first, the catalog refuses, and the writer rebuilds its commit on top of the newer version and tries again. Readers just load whatever the pointer says. Most catalogs now speak the same Iceberg REST protocol, so Spark, Trino, Flink, DuckDB and PyIceberg can share one catalog. A REST catalog can also hand each client short-lived storage credentials scoped to one table’s folder (credential vending), so the catalog’s grants decide who can read the files and engines never hold bucket-wide keys. Catalogs differ in how much they own: for an external table the catalog only records a location you chose, and dropping it leaves the files; for a managed table the catalog picks the location, maintains the files and deletes them when the table is dropped.

Think of it as

A library’s front desk. The books sit on open shelves, but the desk knows which edition of each title is the current one, stamps a new edition in only if nobody beat you to it, and gives each reader a pass that opens just the shelves they’re allowed to browse.

Reach for it when

  • More than one engine reads or writes the same open-format tables, such as Spark for ETL, Trino for dashboards and DuckDB in notebooks
  • You want one place to grant and revoke table access, enforced by vended credentials instead of per-engine copies of the rules and bucket keys
  • Several writers commit to the same tables and need conflicts caught instead of silently overwriting each other
  • You’d rather let the platform choose file locations, compact files and clean up after drops (managed tables)

Think twice when

  • One tool reads a few Parquet files; a catalog adds a service, a database behind it and an auth setup for nothing
  • Engines that bypass the catalog with their own bucket credentials; its grants then protect nothing, so lock down the storage first
  • Expecting the catalog to keep tables tidy: Polaris runs no maintenance, Lakekeeper’s snapshot expiry and orphan-file removal come only in its paid Lakekeeper+ edition and are off by default, and neither compacts files, so plan your own jobs

A real example

A retailer keeps 400 Iceberg tables in one S3 bucket. Spark jobs write them every 15 minutes, analysts query them from Trino, and data scientists open them in DuckDB. All three point at one REST catalog. When an analyst runs a query on sales.orders, Trino asks the catalog for the table, gets the location of its current metadata file plus S3 credentials that only cover that table’s prefix and expire within an hour, and reads the files directly. Two Spark jobs that try to commit to the same table in the same second can’t both win: the catalog accepts one and the other retries on top of it. Revoking the analysts’ grant stops their next table load in every engine at once; credentials already handed out keep working until they expire.

Technologies

Specific products, and what each does differently.

Apache PolarisOpen sourceAn open-source Iceberg REST catalog: one server that every engine asks where a table’s current metadata is, with role-based access control and short-lived storage credentials per table.Cloudflare Basin CatalogRenamedCommercialCloudflare’s managed Apache Iceberg REST catalog built into R2 buckets, called R2 Data Catalog until it became generally available as Basin Catalog on 1 October 2026, which also compacts tables and expires old snapshots for you.LakekeeperOpen sourceAn open-source Apache Iceberg REST catalog written in Rust: a stateless server, backed by PostgreSQL, that tells every engine where each table’s current metadata is, checks permissions, and hands out short-lived storage credentials.Project NessieOpen sourceAn open-source catalog for Apache Iceberg with Git-like branches, tags and commits that span every table at once, so a set of changes to many tables can be tested on a branch and then merged in one atomic step.Unity CatalogCommercialDatabricks’ catalog and governance layer for tables, files, functions and models, with managed and external tables, an Iceberg REST endpoint and credential vending; a separate Apache 2.0 open-source server lives at LF AI & Data.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.Mainly Table formatsAmazon S3 TablesAWSA bucket type for Apache Iceberg tables: S3 stores them, acts as their catalog, and compacts and cleans them up for you.Mainly Lakehouses

What each provider calls it

AWS
AWS Glue Data Catalog (Iceberg REST endpoint), Amazon S3 Tables (table buckets), AWS Lake Formation for grants and credential vending
Google Cloud
Lakehouse runtime catalog (formerly BigLake metastore), with an Iceberg REST endpoint
Azure
Unity Catalog in Azure Databricks; OneLake table APIs in Microsoft Fabric (Iceberg REST, read-only)
Open source & others
Apache Polaris, Lakekeeper, Unity Catalog OSS, Project Nessie, Apache Gravitino, Hive Metastore

Managed and external tables#

The same split exists in Hive, Unity Catalog and most other catalogs. The question is who owns the files’ location and lifecycle; the files stay in your storage either way.

Managed tableExternal table
Who picks the locationThe catalog, under a storage root set on the catalog, schema or warehouseYou, with a LOCATION clause or path that the catalog records
Who may change the filesOnly the catalog’s own write path; Hive warns that changing them behind its back can cause undefined behavior, and Unity Catalog doesn’t allow path-based accessYou and any tool with access to the path; Unity Catalog doesn’t enforce its grants on direct file access
What `DROP TABLE` doesDeletes the data too. Hive moves it to the trash unless you add PURGE; Databricks deletes the files after a recovery period (7 days by default) during which UNDROP brings the table backRemoves only the catalog entry; the files stay until you delete them
MaintenanceThe platform can do it: Databricks’ predictive optimization runs OPTIMIZE, VACUUM and ANALYZE; S3 Tables compacts and expires snapshots by defaultYours: compaction, snapshot expiry and orphan-file cleanup are your jobs
FormatsWhat the catalog can manage: Delta Lake or Iceberg in Unity Catalog, Iceberg in S3 TablesAnything the engines can read: in Unity Catalog also CSV, JSON, Avro, Parquet, ORC and text
Pick it whenThe catalog’s engines are the main writers and you want optimization and cleanup handledOther systems already write the files at a fixed path, or the format isn’t one the catalog manages
  • Databricks recommends managed tables by default and offers ALTER TABLE … SET MANAGED to convert external Delta tables in place. Its other docs page gives the drop retention as 8 days; the managed-tables page says 7 days, configurable to 0 or 7 to 30 days.

The compare-and-swap on the wire#

An Iceberg REST commit carries requirements and updates. Here a writer asserts that main still points at the snapshot it started from, after another writer has already moved it. Run against Lakekeeper 0.14.0; the catalog refuses with 409, which tells the client to reload and retry.

http
POST /catalog/v1/{warehouse-id}/namespaces/sales/tables/orders HTTP/1.1
Content-Type: application/json

{"requirements": [{"type": "assert-ref-snapshot-id", "ref": "main",
                   "snapshot-id": 2973954185555930408}],
 "updates": [{"action": "set-properties", "updates": {"owner": "finance"}}]}

HTTP/1.1 409 Conflict
Content-Type: application/json

{"error": {"type": "CatalogCommitConflicts", "code": 409,
  "message": "CatalogCommitConflicts, context: { expected: 2973954185555930408, found: 2919499479758239524 } => Requirement failed: Branch or tag `main`'s snapshot has changed"}}

Catalogs you’ll meet#

Most of these answer the Iceberg REST protocol, so an engine configured for one can usually talk to another.

Hive Metastore

the ancestor

Hive’s system catalog, described in 2009: databases as namespaces, and for each table its columns, location and format, usually kept in a relational database. Iceberg’s HiveCatalog stores the current metadata file as the table property metadata_location, and many catalogs still offer a Hive-compatible API.

AWS Glue Data Catalog

AWS

AWS’s central metadata store for Athena, EMR and Glue, with an Iceberg REST endpoint at https://glue.<region>.amazonaws.com/iceberg. Lake Formation adds grants and vends temporary S3 credentials.

Amazon S3 Tables

AWS, managed

A bucket type that is its own Iceberg catalog: S3 picks the file locations and runs compaction, snapshot management and unreferenced-file removal.

Lakehouse runtime catalog

Google Cloud

A serverless metastore for Iceberg tables in Cloud Storage, called BigLake metastore until 20 April 2026. It speaks Iceberg REST and vends short-lived tokens.

Unity Catalog

Databricks and LF AI & Data

Databricks’ governance layer, with managed and external tables and volumes, an Iceberg REST endpoint and credential vending. An Apache 2.0 open-source version is a sandbox project at LF AI & Data.

Apache Polaris

Apache, Java

A full Iceberg REST catalog with role-based access control and credential vending, built by Snowflake and a top-level Apache project since February 2026.

Lakekeeper

Apache 2.0, Rust

An Iceberg REST catalog shipped as a single Rust binary, with OpenFGA authorization, credential vending or remote signing, and change events. Maintained by Vakamo.

Project Nessie and Apache Gravitino

open source

Nessie is a transactional catalog with Git-like branches. Gravitino federates metadata from many sources and runs an Iceberg REST service; it became a top-level Apache project in June 2025.

In code · python

# pip install "pyiceberg[pyarrow]"; a Lakekeeper catalog on localhost:8181 with a warehouse "demo"
import pyarrow as pa
from pyiceberg.catalog import load_catalog

catalog = load_catalog("lake", type="rest",
                       uri="http://localhost:8181/catalog", warehouse="demo")
catalog.create_namespace("sales")

schema = pa.schema([("id", pa.int64()), ("amount", pa.float64())])
orders = catalog.create_table("sales.orders", schema=schema)

def pointer(table):  # the metadata file the catalog says is current
    return table.metadata_location.rsplit("/", 1)[-1][:5]

print(pointer(orders))                      # 00000
orders.append(pa.table({"id": [1, 2], "amount": [9.5, 20.0]}, schema=schema))
print(pointer(orders))                      # 00001

# The catalog handed out short-lived storage credentials with the table
print("s3.session-token" in orders.io.properties)   # True

# Two writers start from version 00001; the catalog accepts one commit at a time
other = catalog.load_table("sales.orders")
orders.append(pa.table({"id": [3], "amount": [5.0]}, schema=schema))
print(pointer(orders))                      # 00002
other.append(pa.table({"id": [4], "amount": [7.0]}, schema=schema))
# logs: Commit failed due to a concurrent update, retrying (1/4) in 109 ms
print(pointer(other))                       # 00003: rebuilt on top of 00002

print(catalog.load_table("sales.orders").scan().to_arrow().num_rows)  # 4

Often confused with

Table formatsLakehousesQuery enginesIdentity & access (IAM)
Where it came from2009–2026

Hive, built at Facebook and described in a 2009 VLDB paper, brought a system catalog to Hadoop. Its metastore held each table’s columns, format and directory, with databases as namespaces, and lived in a relational database such as MySQL, or a local file system, because HDFS was built for sequential scans, not small random updates. The catalog said where a table lived, but the files in its directories were the truth, and Hive’s paper admitted it had to keep metadata and data consistent by hand. Hive also splits tables into managed ones, whose files it owns and deletes on DROP, and external ones, whose files it only points at.

Iceberg turned the catalog into the commit point. Its spec keeps a table’s whole state in metadata files and makes a commit one atomic swap of a pointer from the old metadata file to the new one, implemented with a check-and-put in a metastore or database. Any store that could do that worked: the Hive Metastore, AWS Glue, a JDBC database, Nessie. Every engine then needed a client for every catalog, so in July 2022 Iceberg 0.14 added a common REST catalog client, with commits sent as changes that the service checks for conflicts.

The REST protocol became the common interface. In June and July 2024 Databricks open-sourced Unity Catalog and Snowflake open-sourced Polaris, and Gravitino entered the Apache Incubator the same June; all three speak Iceberg REST. The clouds offer their own Iceberg REST endpoints too: AWS Glue and S3 Tables, Google’s BigLake metastore (renamed the Lakehouse runtime catalog in April 2026) and Microsoft’s OneLake. With the formats open, the fight moved to the catalog: who governs access, who maintains the files, and whose engines can write.

Why it’s built this way

Keep the catalog small: one pointer per table
Object stores can’t atomically replace a file or list a directory consistently, but a database can update one row with a check. Putting only the pointer in the catalog keeps the data and the full metadata in open files that any engine can read.
Optimistic commits instead of locks
Writers prepare everything first and only ask the catalog to swap at the end. A conflict costs a retry, not a lock that a crashed job could hold forever, and readers are never blocked.
A REST protocol instead of a client per catalog
The Hive, Glue, JDBC and Nessie catalogs each needed their own client code in every engine. One HTTP protocol means one client per engine, with commits sent as changes and conflicts resolved on the service side.
Vend credentials from the catalog
Grants in a catalog mean little if every engine holds keys to the whole bucket. Returning short-lived credentials scoped to one table with each load lets one set of rules govern every engine.

Timeline

  1. 2009

    Hive paper describes the metastore: databases, tables, locations in a relational database

  2. 2018

    Iceberg enters the Apache Incubator; commits swap a metadata pointer in a catalog

  3. 2021

    Databricks announces Unity Catalog to replace per-workspace Hive metastores

  4. 2022

    Iceberg 0.14 adds a common REST catalog client in July

  5. 2024

    Unity Catalog open-sourced in June, Polaris in July; Gravitino enters the Apache Incubator

  6. 2025

    Gravitino becomes a top-level Apache project; Databricks previews managed Iceberg tables

  7. 2026

    Polaris graduates in February; BigLake metastore becomes the Lakehouse runtime catalog in April

Sources30

Improve this page on GitHub