cloudparts

LakeSoul

A lakehouse table format, first built by DMetaSoul, that keeps all metadata and commits in PostgreSQL and stores primary-key tables as hash buckets with merge-on-read upserts, aimed at CDC ingestion and frequent updates.

LF AI & Data · Sandbox
What it is
Table formats
Used for
Lakehouses
Type
Open source
Made by
LF AI & Data Foundation
Foundation
LF AI & Data Foundation, sandbox, joined May 2023. Part of the Linux Foundation
License
Apache 2.0
Facts checked
23 October 2026
Website Source code

What it is

LakeSoul keeps its catalog in a PostgreSQL database instead of metadata files: tables, partitions, the list of data files in each version and every commit. A writer commits by running a two-stage protocol inside PostgreSQL transactions, which also detect conflicting writers and give readers a consistent snapshot. PostgreSQL triggers notify LakeSoul’s background services when something changes, so compaction and cleanup of expired data start on their own, and PostgreSQL roles and row-level security separate workspaces. The data files are Parquet or, since 4.0, Vortex (the default for new writes), on S3 or HDFS.

A table can have range partitions, laid out in directories like Hive, and a primary key. A primary-key table is split into a fixed number of hash buckets, each sorted by key. An upsert writes only the changed rows, or even only the changed columns, as new delta files, and readers merge the latest value for each key on the fly, much like an LSM tree, until compaction folds the deltas in. That makes many concurrent writers and column-level updates cheap, and lets joins on the key skip the shuffle. A Rust I/O layer, with Java, Python and C bindings, is shared by the Spark 3.5 and Flink 1.20 connectors (read and write), Presto and DuckDB (read), and Python clients for PyArrow, Pandas, Ray and Daft. Flink jobs can sync a whole MySQL database into LakeSoul with automatic table creation and schema changes, and read tables back as changelog streams. DMetaSoul created LakeSoul and donated it to the LF AI & Data Foundation, where it has been a sandbox project since May 2023; the current release is 4.0.0 (September 2026).

Why teams pick it

  • Built for updates: primary-key tables take row- and column-level upserts as small delta files, and concurrent writers to different columns don’t conflict
  • Metadata in PostgreSQL: commits, partition pruning and conflict checks are indexed SQL lookups; the project reports filtering a table with millions of partitions in about 50 ms
  • CDC built in: one Flink job syncs a whole database with new tables and DDL changes picked up automatically, exactly once, and Flink reads tables back as changelogs
  • Table services included: automatic compaction triggered by the metadata database, cleanup of expired and redundant data, and storage statistics

Watch out for

  • Every engine and job needs a connection to the PostgreSQL metadata database, which you run, size and secure; it is the catalog for every table
  • Engines need LakeSoul’s own connectors: Spark 3.5 and Flink 1.20 read and write, Presto and DuckDB only read, and Spark 4 and Flink 2 are on the 2026 roadmap
  • Spark SQL has gaps on primary-key tables: no INSERT INTO (use MERGE INTO), MERGE INTO without conditional MATCHED clauses and only on primary-key equality, and no column type changes
  • A smaller ecosystem than Iceberg, Delta Lake or Hudi, and still at the foundation’s first, sandbox stage

Specs

Compare with alternatives →
Data model
Range partitions and hash buckets sorted by primary key; upserts are delta files merged on read
Transactions
Two-stage commits inside PostgreSQL transactions, which detect conflicting writers
Storage format
Parquet or Vortex (the default since 4.0) on S3 or HDFS
Written in
Rust I/O layer with Java, Python and C bindings
Ways to run it
Needs a PostgreSQL metadata database; Spark and Flink write, Presto and DuckDB read

Alternatives

What all table formats have in common →Compare side by side →
Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.Open Data FabricExperimentalOpen sourceAn open spec for datasets that keep their full history: Parquet files plus a hash-linked chain of metadata blocks, derived data made by deterministic streaming queries, and protocols for sharing it all.

How it fits with other technologies

A primary-key table with hash buckets, and an upsert, in Spark SQL · sql

-- spark.sql.catalog.lakesoul = org.apache.spark.sql.lakesoul.catalog.LakeSoulCatalog
CREATE TABLE default.orders (id string, date string, data string) USING lakesoul
  PARTITIONED BY (date)
  LOCATION 's3://bucket/orders'
  TBLPROPERTIES ('hashPartitions' = 'id', 'hashBucketNum' = '2');

-- primary-key tables take upserts through MERGE INTO, not INSERT INTO
MERGE INTO default.orders AS t USING order_changes AS s
  ON t.id = s.id
  WHEN MATCHED THEN UPDATE SET *
  WHEN NOT MATCHED THEN INSERT *;

See it in a real stack

No architecture uses LakeSoul yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2021–2026

When LakeSoul appeared, Delta Lake and Hudi kept table metadata in files next to the data. Its first README argued that this made metadata slow to parse after a table had run for a long time, that Delta Lake could only be written from a single point, and that Hudi’s merge-on-read didn’t meet its update-performance needs for primary-key data. The DMetaSoul team built LakeSoul on Spark as “a unified streaming and batch table storage solution” with metadata in Cassandra and upserts stored as delta files.

Version 2.0 in July 2022 replaced Cassandra with PostgreSQL and used its transactions for commit conflict detection, and also added snapshot reads and rollback. Version 2.1 brought Flink and whole-database CDC sync. In May 2023 DMetaSoul donated LakeSoul to the LF AI & Data Foundation as a sandbox project, and 2.3.0 that July was the first release under the foundation, with Flink SQL and global automatic compaction.

LakeSoul 3.0 in September 2025 doubled read and write speed of its Rust I/O layer compared with 2.6, added a local cache for object storage, Spark with Gluten and Presto with Velox, an Arrow Flight SQL gateway and a PyPI package. Version 4.0 followed in September 2026, with Vortex as the default file format for new writes.

Why it’s built this way

Metadata in a database, not in files
Its first README blamed file-based metadata for slow metadata parsing on long-running tables and for single-point writes. A database handles many concurrent commits and indexed lookups; LakeSoul started on Cassandra and moved to PostgreSQL in 2.0 to get transactions for conflict detection.
Upserts as delta files, merged when read
Writing only the changed rows or columns turns updates into appends, which the project says greatly improves write throughput and concurrency; compaction merges the deltas later.
Hash buckets sorted by primary key
Records with the same key always land in the same sorted bucket, so a read can merge versions without a shuffle, and joins, intersects and excepts on the key need no shuffle or sort.
Let the metadata database drive maintenance
Compaction, cleanup and statistics services listen to change notifications from PostgreSQL, so they run cluster-wide without configuring each table, which 2.3 introduced to cut the cost of keeping many tables fast.

Timeline

  1. 2021

    DMetaSoul publishes LakeSoul on Spark, with metadata in Cassandra

  2. 2022

    2.0 moves metadata to PostgreSQL (July); 2.1 adds Flink CDC sync (October)

  3. 2023

    Donated to LF AI & Data as a sandbox project (May); 2.3 adds Flink SQL and global compaction (July)

  4. 2025

    3.0 in September: faster Rust I/O, Gluten and Presto with Velox, PyPI package

  5. 2026

    4.0 in September, with Vortex as the default file format

Sources8 · facts checked 23 October 2026

Improve this page on GitHub