Apache Paimon

A lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.

What it is: Table formats
Used for: Lakehouses
Type: Open source
Made by: Apache Software Foundation
License: Apache 2.0
Facts checked: 6 October 2026

What it is

Paimon keeps a table as files under one directory, like Iceberg: a snapshot file (JSON) names the schema and a manifest list, manifests list the data files, and data files are Parquet by default, or ORC or Avro. What differs is how rows with a primary key are stored. Each partition is split into buckets by a hash of the key, and each bucket is an LSM tree: writers buffer records in memory, sort them by key and flush them as a new sorted run. Readers merge the runs, and a background compaction, similar to RocksDB’s universal compaction, folds runs together so reads stay fast. When two records share a key, a merge engine decides the result: keep the latest (the default), fill in non-null columns from several streams (partial update), aggregate, or keep the first row.

Because every commit is a snapshot, a streaming job can read a table as an endless queue of changes and pass them to the next table, which is how Flink pipelines chain Paimon tables. A changelog producer controls whether those changes include old values: input stores the incoming change stream, while lookup and full-compaction compute it during compaction. Flink and Spark read and write, including streaming; Hive, Trino, Presto, StarRocks and Doris can read, and Paimon can also write Iceberg-compatible metadata so Iceberg readers see the same files. Tables can sit in HDFS, S3, Azure, Google Cloud Storage and other stores, with the catalog in the file system, Hive Metastore, a JDBC database or a REST service. Paimon 2.0, from August 2026, adds multimodal tables with BLOB, vector and variant columns and vector and full-text indexes.

Why teams pick it

High-rate upserts and deletes into object storage: writes are appends to an LSM tree rather than rewrites of whole files
A table doubles as a change stream, so streaming jobs can chain table to table instead of through Kafka topics
Merge engines handle common streaming patterns in storage: wide tables assembled from several streams (partial update) and running aggregates
Built-in CDC jobs sync MySQL, PostgreSQL, MongoDB, Kafka or Pulsar sources into tables, following schema changes
Iceberg-compatible metadata lets engines that only speak Iceberg read Paimon tables

Watch out for

Streaming writes and reads need Flink, or Spark in micro-batches; Hive, Trino, Presto, StarRocks and Doris only read or batch-write
Bucket layout needs care: too many buckets make small files, too few limit write parallelism, and changing a fixed bucket count is an offline job
Dynamic buckets, the default for primary-key tables, allow only one writing job per partition; a second job can duplicate data
Compaction competes with writes, and only one job may compact a partition, so several writers to one table need a separate compaction job
Changelog producers slow compaction, and the docs advise enabling them only when a downstream consumer needs old values

Alternatives

What all table formats have in common →

Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.

Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.

Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.

DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.

LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.

How it fits with other technologies

What Apache Paimon uses

What uses Apache Paimon

A primary-key table on S3, read as a stream and as of a snapshot (Flink SQL) · sql

-- paimon-flink and paimon-s3 jars in Flink's lib directory
CREATE CATALOG lake WITH (
  'type' = 'paimon',
  'warehouse' = 's3://acme-lake/paimon',
  's3.endpoint' = 's3.eu-west-1.amazonaws.com',
  's3.access-key' = '...',
  's3.secret-key' = '...'
);
USE CATALOG lake;

-- each bucket of each partition is an LSM tree sorted by the key
CREATE TABLE orders (
  order_id BIGINT,
  status STRING,
  amount DECIMAL(10, 2),
  updated_at TIMESTAMP(3),
  dt STRING,
  PRIMARY KEY (dt, order_id) NOT ENFORCED
) PARTITIONED BY (dt) WITH (
  'sequence.field' = 'updated_at',   -- newest update wins, even out of order
  'changelog-producer' = 'lookup'    -- emit old and new values for consumers
);

SET 'execution.runtime-mode' = 'streaming';
-- another job: read only new changes, as an unbounded stream
SELECT * FROM orders /*+ OPTIONS('scan.mode' = 'latest') */;

-- batch: the table as of snapshot 42
SELECT * FROM orders /*+ OPTIONS('scan.snapshot-id' = '42') */;

Paimon, Iceberg, Hudi or Delta Lake?#

All four turn files in object storage into tables with snapshots. They differ in how they apply updates by key and how well a downstream job can follow the changes.

	Paimon	Iceberg	Hudi	Delta Lake
How commits are tracked	Snapshot files pointing to manifest lists and manifests	Metadata file pointing to snapshots, manifest lists and manifests; a catalog swaps the pointer	A timeline of commits, compactions and cleans in `.hoodie`	Numbered JSON commits in `_delta_log`, with Parquet checkpoints
Updates by key	LSM tree per bucket; merged at read and in compaction, with a choice of merge engines	Rewrite files, or write delete files or deletion vectors	Index finds the file group; copy-on-write or log files merged later	Rewrite files, or deletion vectors
Following changes downstream	Streaming read of every snapshot; changelog producers add old values	Streaming reads process append snapshots only; overwrites and deletes fail or are skipped	Incremental queries return records changed since a commit	Change data feed, when enabled on the table

Paimon was designed for Flink pipelines that update tables continuously. It can also write Iceberg metadata, as Delta Lake can through UniForm, so engines that only read Iceberg can query the same files.

See it in a real stack

No architecture uses Apache Paimon yet. Drop it into the playground and wire it up yourself.Open playground →

Worth exploring

Article· September 2025Understanding Apache FlussJack VanlightlyA long, detailed walk through Fluss internals, reverse-engineered from the code: log tables, primary-key tables that emit changelogs, and tiering to lakehouse tables. The author works at Confluent, which runs Flink but not Fluss, and says so up front.

All resources →

Where it came from2022–2026

Flink users building real-time pipelines typically wired up several stores: a message queue such as Kafka between streaming jobs, a file system or object store for batch and ad-hoc queries, and a key-value store for lookups. Intermediate Kafka tables couldn’t be queried, so teams dual-wrote to other systems, and the lake formats of the time, Hudi and Iceberg, struggled to take a high-speed stream of updates.

Jingsong Lee at Alibaba proposed FLIP-188, a storage layer built into Flink that combined a log for streaming with an LSM-based file store in columnar formats. It shipped as Flink Table Store 0.1.0, a preview, in May 2022. As engines other than Flink started using it, users questioned whether a Flink sub-project could be neutral, so it left Flink and entered the Apache Incubator as Paimon in March 2023, with developers from Alibaba, ByteDance, Bilibili and Confluent. Paimon 0.4 in June 2023 was the first release under the new name.

Paimon became a top-level Apache project in March 2024 and released 1.0 in early 2025, after production use at Alibaba and ByteDance. Version 2.0 in August 2026 turned toward AI data, with BLOB and vector columns and indexes in the same tables.

Why it’s built this way

An LSM tree inside each bucket: LSM trees turn updates and deletes into appends, which suits an object store that can’t edit files in place and a stream that never stops writing. Compaction merges the sorted runs later.
Changes as a first-class output: The goal was to replace the queue between streaming jobs. Storing snapshots and, optionally, full changelogs lets a downstream Flink job read the table as a stream with old and new values.
Leaving Flink for an independent project: As Spark, Hive and others adopted it, users worried about the neutrality of a storage layer owned by one engine’s community, so it moved to its own Apache project and name.

Timeline

2022
Flink Table Store 0.1.0 preview released in May
2023
Enters the Apache Incubator as Paimon in March; 0.4 in June
2024
Graduates to a top-level Apache project in March
2025
Paimon 1.0, a stabilisation release
2026
Paimon 2.0 adds multimodal tables, vectors and search indexes

Sources23 · facts checked 6 October 2026

Improve this page on GitHub