Cloud Parts

Apache Hudi

A table format built for upserts and incremental pulls, with record-level indexes and built-in table services.

What it is
Table formats
Used for
Lakehouses
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
5 October 2026
Website Source code

What it is

Uber built Hudi in 2016 to apply database changes to its Hadoop data lake within minutes instead of rewriting whole partitions every night. Every record has a key. On each write, Hudi looks the incoming keys up in an index to find the file group that holds them, then either rewrites that file (copy-on-write) or appends the changes to a row-based log file next to it (merge-on-read). A later compaction folds the logs into new Parquet base files. A timeline in the table’s .hoodie folder records every commit, compaction and clean.

Incremental queries are the other half: a downstream job asks for “every record changed since commit X” and gets only those rows, so pipelines can chain table to table without rescanning. Table services for compaction, clustering and cleaning can run inline with writes or as separate jobs. Uber open-sourced Hudi in 2017 and gave it to the Apache Software Foundation in 2019; the current release is 1.2.1.

Why teams pick it

  • Efficient upserts and deletes by record key, which suits change-data capture from databases
  • Incremental queries turn any table into a change stream for the next job
  • Merge-on-read keeps write latency low for tables with heavy updates
  • Ingestion, compaction, clustering and cleaning services come built in

Watch out for

  • More knobs than Iceberg or Delta Lake: table type, index type and compaction and cleaning policies all need choosing
  • Engine support is narrower: Trino’s connector is read-only and sees merge-on-read tables only as of the last compaction
  • Merge-on-read queries slow down when compaction falls behind
  • Writes are mostly Spark or Flink jobs

Alternatives

What all table formats have in common →
Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.

How it fits with other technologies

A merge-on-read table and an incremental read (Spark SQL) · sql

CREATE TABLE orders (
  order_id BIGINT, amount DOUBLE, updated_at BIGINT, order_date STRING
) USING HUDI
TBLPROPERTIES (type = 'mor', primaryKey = 'order_id', orderingFields = 'updated_at')
PARTITIONED BY (order_date)
LOCATION 's3://acme-lake/orders';

MERGE INTO orders t USING order_changes c ON t.order_id = c.order_id
WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *;

-- only rows changed since a given commit
SELECT * FROM hudi_table_changes('orders', 'latest_state', '20261004000000');

See it in a real stack

No architecture uses Apache Hudi yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2016–2026

By 2016 Uber kept petabytes of data in Hadoop, and much of it changed constantly. HDFS files can’t be edited, so applying updates meant batch jobs that rewrote whole partitions every few hours, and late-arriving data forced rewrites of older partitions too. The usual alternative, a lambda architecture, ran a separate streaming layer next to the batch one, and its approximate results drifted from the batch numbers. Fast serving stores could take record-level updates, but they cost more to scale and usually kept only 30 to 90 days or a few terabytes of data.

Uber engineers including Vinoth Chandar and Prasanna Rajaperumal built Hudi in 2016 under the codename “Hoodie”. It applied changes to HDFS tables within minutes and let downstream jobs pull only the records that had changed. Uber described it publicly in March 2017, open-sourced it that year, and submitted it to the Apache Incubator in January 2019. By then it managed over 4,000 tables and several petabytes at Uber, and had cut warehouse data latency from several hours to under 30 minutes. Hudi graduated to a top-level Apache project in May 2020.

Recent releases have widened its scope: 1.0 in December 2024 added secondary indexes and non-blocking concurrency control, and 1.2 in June 2026 added vector search and Lance files for AI data.

Why it’s built this way

Record keys and a pluggable index
To update a record, Hudi must know which file holds it. An index (Bloom filters by default, HBase as an option) maps incoming keys to file groups, so only those files are touched.
Avro logs for writes, Parquet for reads
Appending changes to row-based Avro log files is cheap; Parquet is fast to scan. Merge-on-read writes to the logs first and compacts them into Parquet later, trading some query cost for low write latency.
A timeline of every commit
Recording each commit, clean and compaction with its time lets a reader pick file versions as of a commit and ask for only the records changed since one, which is what makes incremental pulls work.
Minutes, not seconds, on plain Hadoop
Rather than add a fast serving store, Uber targeted workloads that could tolerate about 10 minutes of latency, keeping data in HDFS where storage is cheap and scans are fast.

Timeline

  1. 2016

    Built at Uber under the codename Hoodie

  2. 2017

    Uber describes it publicly in March and open-sources it

  3. 2019

    Enters the Apache Incubator in January

  4. 2020

    Graduates to a top-level Apache project in May

  5. 2024

    Hudi 1.0 adds secondary indexes and non-blocking concurrency control

  6. 2026

    Hudi 1.2 adds vector search, variant, blobs and Lance file support

Sources11 · facts checked 5 October 2026

Improve this page on GitHub