Delta Lake

An open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.

What it is: Table formats
Used for: Lakehouses
Type: Open source
Made by: Linux Foundation (created by Databricks)
License: Apache 2.0
Facts checked: 5 October 2026

What it is

A Delta table is a folder of Parquet files plus a _delta_log directory. Each commit adds the next numbered JSON file to the log, listing which data files were added and removed. Every so often the log is summarised into a Parquet checkpoint, so readers don’t have to replay thousands of JSON files. Writers use optimistic concurrency: two writers race to create the same log entry, one wins, and the other checks for conflicts and retries.

Because the log sits next to the data, the folder is the table: any engine that can read the log can read it, with or without a catalog. Delta Lake was created at Databricks and is now a Linux Foundation project. UniForm (Universal Format) also writes Iceberg metadata, and Hudi metadata in preview, for the same data files, so Iceberg clients can read a Delta table without a copy. Those clients can only read; writes still go through Delta.

Why teams pick it

Deep Spark integration: MERGE, streaming reads and writes on one table, and a change data feed
No catalog needed to read a table; the log in the folder is the source of truth
Deletion vectors (delete rows without rewriting files) and liquid clustering for layout
UniForm lets Iceberg readers query the same files

Watch out for

Support outside Spark and Databricks is uneven: Athena, for example, reads Delta tables but can’t write them or time travel
Concurrent writers from several clusters on S3 need a DynamoDB-backed log store, because S3 alone can’t guarantee only one writer creates a log entry
UniForm is read-only for Iceberg clients and can’t be used together with deletion vectors
Frequent small commits grow the log and the file count; OPTIMIZE and VACUUM need to run regularly

Alternatives

What all table formats have in common →

Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.

Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.

Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.

DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.

LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.

How it fits with other technologies

What Delta Lake uses

What uses Delta Lake

A Delta table with UniForm on, then history and time travel (Spark SQL) · sql

CREATE TABLE sales.orders (order_id BIGINT, amount DECIMAL(10,2), ts TIMESTAMP)
USING DELTA
TBLPROPERTIES (
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

DESCRIBE HISTORY sales.orders;
SELECT count(*) FROM sales.orders VERSION AS OF 3;

OPTIMIZE sales.orders;
VACUUM sales.orders RETAIN 168 HOURS;

See it in a real stack

No architecture uses Delta Lake yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2017–2026

Databricks’ customers kept their tables in cloud object stores like S3 as folders of Parquet files. That was cheap, but object stores are key-value stores with no multi-object transactions. An update that touched many files let readers see partial results, a crashed job left a table corrupted, and listing millions of objects was slow. In the first years of Databricks’ cloud service (2014–2016), around half of its support escalations came from data corruption, consistency or performance problems of this kind.

A Databricks team including Michael Armbrust, Tathagata Das and Reynold Xin built Delta as an ACID storage layer over object stores and started offering it to customers in October 2017. Databricks open-sourced it under Apache 2.0 on 24 April 2019, when Viacom, Edmunds, Riot Games and McGraw Hill already used it, and moved it to the Linux Foundation that October. A 2020 VLDB paper described the design, and in 2022 Databricks committed to open-sourcing all of Delta’s features with Delta Lake 2.0.

The same paper argued that Delta enabled a “lakehouse”: warehouse features directly on cheap object storage. Iceberg grew into a strong rival, and the two moved toward each other: UniForm (2023) writes Iceberg metadata for Delta tables, and in June 2024 Databricks agreed to acquire Tabular, the company founded by Iceberg’s creators.

Why it’s built this way

The transaction log lives in the table’s folder: The list of files in each version is kept as a write-ahead log stored in the object store itself, with the data in Parquet, so any engine that already reads Parquet can learn to read a Delta table.
Commit by creating the next numbered log file: Only one writer may create 000004.json. Azure and Google Cloud Storage can do an atomic put-if-absent and HDFS an atomic rename, but S3 could do neither, so Databricks used a separate coordination service for S3 log writes.
Checkpoints that compact the log into Parquet: Replaying thousands of small JSON records is slow on high-latency object stores. A Parquet checkpoint lets readers load table state and file statistics quickly, even across billions of partitions.
One table for streaming and batch: Streaming jobs write small files at low latency and later coalesce them in a transaction, and readers can tail new commits, so a Delta table can act as a message bus as well as a batch table.

Timeline

2017
Databricks launches Delta for its customers in October
2019
Open-sourced in April; moves to the Linux Foundation in October
2022
Delta Lake 2.0; Databricks commits to open-sourcing all Delta features
2023
Delta Lake 3.0 introduces UniForm for Iceberg and Hudi readers
2024
Databricks agrees to acquire Tabular, founded by Iceberg’s creators
2025
Delta Lake 4.0 adds variant, row tracking and catalog-managed tables (preview)
2026
Delta Lake 4.1 makes catalog-managed tables production-ready

Sources14 · facts checked 5 October 2026

Improve this page on GitHub