Cloud Parts

Unity Catalog

Databricks’ catalog and governance layer for tables, files, functions and models, with managed and external tables, an Iceberg REST endpoint and credential vending; a separate Apache 2.0 open-source server lives at LF AI & Data.

What it is
Lakehouse catalogs
Used for
Lakehouses
Type
Commercial product
Made by
Databricks, Inc.
License
Proprietary as a Databricks service; the open-source Unity Catalog server is Apache 2.0
Facts checked
9 October 2026
Website Source code

What it is

The name covers two things. Most people meet Unity Catalog as the governance layer built into Databricks, on by default for workspaces created after 8 November 2023. There, one metastore per cloud region is shared by the workspaces attached to it, so a grant made in one workspace applies in all of them. Everything you govern is a securable object in a three-level namespace, catalog.schema.object: tables, views, volumes (governed folders of non-tabular files under /Volumes/<catalog>/<schema>/<volume>/), functions, models and services. Storage credentials, external locations, connections and shares sit directly under the metastore. Access is controlled with GRANT and ownership, attribute-based policies on governed tags, per-table row filters and column masks, and workspace bindings, and Unity Catalog records lineage and an audit log as data is used.

Tables and volumes come in two kinds. Managed tables are Databricks’ default and recommendation: Unity Catalog picks where the files go inside a managed storage location in your own cloud account (set on the metastore, catalog or schema), stores them as Delta Lake or, with USING iceberg, as Iceberg, and looks after them. Predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on serverless compute, automatic liquid clustering picks clustering keys from your queries, and a dropped managed table’s files are deleted after a recovery period of 7 days by default, during which UNDROP brings it back. External tables point at a path you choose inside a registered external location, can also be CSV, JSON, Avro, Parquet, ORC or text, get no automatic optimization, and keep their files when dropped. Foreign tables mirror tables that another catalog, such as a Hive Metastore, AWS Glue or Snowflake Horizon Catalog, still owns, and are read-only. Other engines connect through two open APIs, the Unity REST API for Delta clients and an Iceberg REST catalog at /api/2.1/unity-catalog/iceberg-rest, which can read and write managed Iceberg tables and read Delta tables that have Iceberg reads enabled. Both vend short-lived storage credentials that carry the requesting principal’s privileges.

The open-source Unity Catalog is a separate, much smaller implementation. Databricks published it on 12 June 2024 under Apache 2.0 and gave it to the LF AI & Data Foundation, where it is still a sandbox project. It is a Java 17 server with a REST API, a CLI and a web UI that keeps its metadata in H2 by default, with PostgreSQL as an option, and it models the same catalogs, schemas, tables, volumes, functions and models. It added OAuth sign-in, privileges and credential vending for S3, Azure and Google Cloud Storage in 0.2 (September 2024), managed storage locations with experimental managed Delta tables in 0.4 (February 2026), a REST API for catalog-managed Delta commits in 0.5, and views in 0.6 (August 2026). Its Iceberg REST endpoint serves Delta tables with UniForm to Iceberg readers, and its README warns that the APIs are still evolving.

Why teams pick it

  • One permission model for tables, views, volumes, functions and models, with tag-based policies, row filters, column masks, lineage and audit logs, shared by every workspace on the metastore
  • Managed tables look after themselves: predictive optimization compacts, vacuums and collects statistics, automatic liquid clustering picks keys, and UNDROP recovers a dropped table within the recovery period
  • Open to other engines: Spark, Flink, Trino, DuckDB and other Iceberg REST or Delta clients can read and write managed tables with vended credentials, instead of only reading copies
  • Governs tables it doesn’t own: catalog federation brings Hive Metastore, Glue and Snowflake Horizon tables under the same grants without moving them, and SET MANAGED converts external Delta tables in place
  • An Apache 2.0 server you can run yourself in Docker, with the same namespace and API shape

Watch out for

  • Most of the features above are Databricks-only. The open-source server is at 0.x as a sandbox project: its README says the APIs shouldn’t be assumed stable, managed Delta tables are experimental, authorization is off in the default config, and its Iceberg endpoint only serves UniForm tables for reading
  • Managed tables must be used by name: path-based access isn’t supported (except through Compatibility Mode), creating and writing managed Delta tables from outside Delta clients is in Public Preview, and external Iceberg engines can’t read Unity Catalog views
  • The automation costs money and setup: predictive optimization needs the Premium plan and bills its runs as serverless jobs, and managed Iceberg tables need serverless compute that can reach your storage
  • External tables are only governed through Unity Catalog: its grants don’t apply to direct file access, and metadata changed by another client must be synced back with MSCK REPAIR TABLE … SYNC METADATA
  • Foreign tables are read-only (except an internal federated Hive Metastore) and don’t get managed tables’ transactional guarantees or most optimizations

Alternatives

What all lakehouse catalogs have in common →
Apache PolarisOpen sourceAn open-source Iceberg REST catalog: one server that every engine asks where a table’s current metadata is, with role-based access control and short-lived storage credentials per table.Cloudflare Basin CatalogRenamedCommercialCloudflare’s managed Apache Iceberg REST catalog built into R2 buckets, called R2 Data Catalog until it became generally available as Basin Catalog on 1 October 2026, which also compacts tables and expires old snapshots for you.LakekeeperOpen sourceAn open-source Apache Iceberg REST catalog written in Rust: a stateless server, backed by PostgreSQL, that tells every engine where each table’s current metadata is, checks permissions, and hands out short-lived storage credentials.Project NessieOpen sourceAn open-source catalog for Apache Iceberg with Git-like branches, tags and commits that span every table at once, so a set of changes to many tables can be tested on a branch and then merged in one atomic step.

How it fits with other technologies

What Unity Catalog uses

The open-source server in Docker: an external table outlives its catalog entry · bash

docker run -d --name uc -p 8080:8080 unitycatalog/unitycatalog:v0.6.0
docker exec -it uc bash          # the CLI ships in the image

bin/uc catalog create --name lake
bin/uc schema create --catalog lake --name sales
bin/uc table create --full_name lake.sales.events \
  --columns "id INT, name STRING" --storage_location /tmp/ext/events
bin/uc table write --full_name lake.sales.events   # sample rows, as a Delta table
bin/uc table get --full_name lake.sales.events --output jsonPretty
#   "table_type" : "EXTERNAL",
#   "data_source_format" : "DELTA",
#   "storage_location" : "file:///tmp/ext/events",
bin/uc table read --full_name lake.sales.events --max_results 2

bin/uc table delete --full_name lake.sales.events
ls /tmp/ext/events       # _delta_log  e96fb6de-…-000.parquet  (files are still there)
exit

# The same server answers Iceberg REST clients
curl -s localhost:8080/api/2.1/unity-catalog/iceberg/v1/catalogs/unity/namespaces
# {"namespaces":[["commerce"],["consumer"],["default"]],"next-page-token":null}

Table types in Databricks Unity Catalog#

From Databricks’ documentation, checked 2026-10-09. Volumes split into managed and external the same way.

ManagedExternalForeign
Where the files liveA managed storage location in your cloud account, chosen by Unity CatalogA path you give, inside a registered external locationWherever the other catalog keeps them
Who owns the metadataUnity CatalogUnity CatalogThe other catalog: Hive Metastore, Glue, Snowflake Horizon or a database reached by query federation
FormatsDelta Lake or IcebergDelta, CSV, JSON, Avro, Parquet, ORC, textWhatever the source holds
WritesDatabricks, and outside engines through the Iceberg REST and Unity REST APIsDatabricks and Delta clients; direct file writes bypass Unity CatalogRead-only, except an internal federated Hive Metastore
Automatic maintenancePredictive optimization, automatic liquid clustering, metadata cachingNoneNone
`DROP TABLE`Files deleted after the recovery period (7 days by default; 0 or 7 to 30 days configurable); UNDROP restoresRemoves the entry; files stayData and metadata stay with the external system that manages them
  • Databricks recommends managed tables, and ALTER TABLE … SET MANAGED converts external or foreign Delta tables to managed ones while keeping their history and permissions.
  • Another Databricks page gives the drop retention for managed tables and volumes as 8 days; the managed-tables page says 7.

Managed and external tables in Databricks SQL#

The table lifecycle from Databricks’ documentation. This needs a Databricks workspace, so unlike the snippet above it wasn’t run for this page.

sql
-- Managed: Unity Catalog chooses the location and maintains the files
CREATE OR REPLACE TABLE main.sales.orders (id BIGINT, amount DECIMAL(10,2))
CLUSTER BY AUTO;
CREATE TABLE main.sales.refunds (id BIGINT, amount DECIMAL(10,2)) USING iceberg;

-- External: you choose the location, inside a registered external location
CREATE TABLE main.raw.clicks (id BIGINT, url STRING)
LOCATION 's3://acme-raw/clicks';

DROP TABLE main.raw.clicks;     -- the files in s3://acme-raw/clicks stay
DROP TABLE main.sales.orders;   -- the files go after the recovery period
UNDROP TABLE main.sales.orders; -- ...unless you undo it in time

ALTER SCHEMA main.sales RETAIN DROPPED TO 30 DAYS;
ALTER TABLE main.raw.events SET MANAGED;  -- convert an external Delta table

Databricks Unity Catalog and Unity Catalog OSS#

Same name and namespace, different software. Checked 2026-10-09.

In DatabricksOpen source
Who runs itDatabricks, inside every Unity Catalog-enabled Databricks workspaceYou: Docker image, tarball or Helm chart
StatusGenerally available since 2022; managed Iceberg tables GA since May 20260.6.0 (August 2026), a sandbox project at LF AI & Data
Managed tablesDelta Lake and Iceberg, with predictive optimizationExperimental managed Delta tables since 0.4
Iceberg RESTReads and writes managed Iceberg tables; reads Delta with Iceberg reads enabledReads Delta tables with UniForm
Access controlPrivileges, tag-based policies, row filters, column masks, workspace bindingsUsers and privileges with an external identity provider; off by default
Credential vendingTables and paths, through the Unity REST and Iceberg REST APIsS3, Azure and Google Cloud Storage; AWS storage credentials and external locations since 0.4

See it in a real stack

No architecture uses Unity Catalog yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2021–2026

Before Unity Catalog, each Databricks workspace had its own Hive metastore and a separate service for table access controls. Companies with many workspaces had to copy table definitions, views and permissions between them by hand, and the copies drifted. Databricks announced Unity Catalog at its Data and AI Summit in 2021 as one governance layer for data and AI, opened a gated public preview on AWS and Azure in April 2022, and made it generally available later that year (on 25 August 2022 in Azure Databricks). It added a catalog level above Hive’s database.table, so the old metastore could show up as one catalog, hive_metastore, next to the new ones. From November 2023, new workspaces got Unity Catalog automatically.

In June 2024, about a week after agreeing to buy Tabular, the company founded by Iceberg’s creators, Databricks open-sourced Unity Catalog. It released version 0.1 under Apache 2.0, hosted at the Linux Foundation’s LF AI & Data, with the Iceberg REST catalog API built with Tabular’s expertise and credential vending to gate access to storage. Databricks argued that most platforms were walled gardens and promised that its hosted service implemented the same open APIs.

On the Databricks side, Unity Catalog then took on Iceberg as a first-class format: managed Iceberg tables that outside engines can write through the Iceberg REST API went into public preview in June 2025 and became generally available in May 2026, along with Iceberg v3 and governed access to Iceberg tables in other catalogs. The open-source server has grown more slowly, through 0.x releases that added authentication, storage credentials, managed storage and views.

Why it’s built this way

Three levels: catalog, schema, object
Hive had only databases and tables. A level on top gave data owners more room to organize data and let the existing Hive metastore appear as one catalog, so customers could adopt Unity Catalog alongside it.
One metastore per region, shared by workspaces
Per-workspace metastores meant hand-synchronised definitions and permissions. A shared metastore makes a grant in one workspace effective in all workspaces attached to it.
Managed tables in the customer’s own account
Unity Catalog decides where managed files go and maintains them, but they stay in the customer’s cloud storage, not Databricks’. That lets Databricks automate layout and cleanup without taking custody of the data.
Open APIs and credential vending
Other engines can’t be trusted to apply Databricks’ grants, so the catalog checks them and returns short-lived credentials scoped to the table. The same idea underpins the open-source release.

Timeline

  1. 2021

    Announced at Data and AI Summit

  2. 2022

    Gated public preview in April; generally available by August (Azure Databricks: 25 August)

  3. 2023

    Enabled automatically for new workspaces from 8 November

  4. 2024

    Open-sourced at LF AI & Data in June; OSS 0.1 in July

  5. 2025

    Managed Iceberg tables and Iceberg REST writes in public preview (June)

  6. 2026

    Managed Iceberg GA in May; OSS 0.6 adds views in August

Sources28 · facts checked 9 October 2026

Improve this page on GitHub