Apache Fluss

Streaming storage built for Flink: event logs and updatable tables in columnar Arrow format, with key lookups, and older data copied into Paimon, Iceberg, Hudi or Lance tables.

What it is: Event streams
Used for: Lakehouses
Type: Open source
Made by: Apache Software Foundation
License: Apache 2.0 (since it was open-sourced in November 2024)
Facts checked: 6 October 2026

What it is

Where Kafka stores topics of opaque records, Fluss stores tables with a schema. A cluster has CoordinatorServers, which keep metadata and decide where data lives (one is active, elected through ZooKeeper), and TabletServers, which hold the data. A table is split into partitions, for example by date, and each partition into buckets. Each bucket is a replicated log with a leader and followers, built on an adapted copy of Kafka’s replication code. In a log table, clients write records as Apache Arrow column batches, so a reader that asks for three columns, or for rows matching a filter, gets only those from the server. A primary-key table adds a RocksDB store on the bucket leader: each upsert or delete is checked against the current row, the resulting change (insert, update before and after, delete) goes into the replicated log, and then RocksDB is updated. That gives you point lookups, partial updates and a ready-made changelog for downstream jobs. TabletServers move older log segments and RocksDB snapshots to S3, HDFS or similar, and clients download those files directly instead of streaming them through the servers.

A separate tiering service, which runs as a Flink job, copies tables into an open lake format (Paimon, Iceberg, Hudi or Lance) every few minutes (3 by default), with exactly-once commits. A query on the table then reads the lake snapshot plus the newer rows still in Fluss, which the project calls a union read. Flink (1.18 to 2.3) is the main engine; Spark 3.4 and 3.5, and Java, Rust, Python and C++ clients, also work, and 1.0 adds an HTTP gateway as a preview. Alibaba’s Flink team started Fluss in 2023 and open-sourced it in November 2024; it graduated to a top-level Apache project in July 2026, and 1.0 followed in September 2026. It is free to run; Alibaba Cloud offers it as a managed service and Ververica includes it in its self-managed Flink platform.

Why teams pick it

Primary-key tables with server-side changelogs, partial updates and lookups move join and aggregation state out of Flink jobs; Flink 2.1+ turns eligible joins into delta joins that read Fluss tables instead of keeping both sides in state
Columnar Arrow logs let the server prune columns and skip batches by filter, so a job that needs a few columns of a wide table doesn’t receive the rest
Lake tiering into Paimon, Iceberg, Hudi or Lance keeps history in open formats other engines can read, and union reads combine it with second-fresh data
Run at large scale at Alibaba (the project reports 3 PB and 40 GB/s peak throughput in 2025) and in production at RedNote, JD.com, Ant Group and others

Watch out for

Built around Flink: the tiering service is a Flink job, union reads are native only in Flink (Spark got batch union reads in 1.0), and other engines read the lake copy
Not a Kafka replacement for event transport: there is no Kafka protocol, so producers and consumers need Fluss clients, connectors or the preview gateway
More to operate: ZooKeeper 3.6 or newer (removal is on the roadmap), local disks with cross-server replication, and RocksDB state that must fit on the bucket leader’s disk and is rebuilt from a snapshot plus log replay when leadership moves
Schema changes are limited to adding nullable columns at the end; renames and type changes are still on the roadmap, and lake tables must never be altered from another engine
Young: 1.0 is from September 2026, the gateway is a preview, and features such as historical partition lookups work only with Paimon

Alternatives

What all event streams have in common →

Amazon Kinesis Data StreamsAWSAWS’s serverless event stream, split into shards that each keep records in order.

Apache KafkaOpen sourceThe open-source distributed log that most event streaming is built on, or built to be compatible with.

Apache PulsarOpen sourceA multi-tenant messaging and streaming platform from Yahoo that keeps brokers stateless and stores messages in Apache BookKeeper, so one topic can serve as a log, a pub-sub feed or a work queue.

Cloudflare K2BetaCommercialCloudflare’s serverless event stream: a durable log stored in R2 that Workers and HTTP clients append to, read through independent subscriptions that lease out batches of records.

RedpandaCommercialA Kafka-compatible streaming broker rewritten in C++ as one binary, with no JVM and no ZooKeeper.

WarpStreamCommercialA Kafka-compatible stream that keeps no data on disks: stateless Agents write straight to your object storage bucket.

How it fits with other technologies

What Apache Fluss uses

What uses Apache Fluss

Apache ZooKeeperelects the active CoordinatorServer forApache FlussCoordination services

Flink SQL: a log table, a primary-key table tiered into the lake, lookups and a union read · sql

CREATE CATALOG fluss WITH (
  'type' = 'fluss',
  'bootstrap.servers' = 'coordinator-server:9123'
);
USE CATALOG fluss;

-- append-only log table, stored as Arrow batches in 8 buckets
CREATE TABLE orders (
  order_id BIGINT,
  customer_id BIGINT,
  amount DECIMAL(10, 2),
  ts TIMESTAMP(3)
) WITH ('bucket.num' = '8');

-- primary-key table, also copied into the lake format set on the cluster
CREATE TABLE customer_totals (
  customer_id BIGINT,
  orders BIGINT,
  revenue DECIMAL(18, 2),
  PRIMARY KEY (customer_id) NOT ENFORCED
) WITH (
  'bucket.num' = '8',
  'table.datalake.enabled' = 'true',
  'table.datalake.freshness' = '30s'
);

-- streaming job: Fluss turns each upsert into a changelog entry
INSERT INTO customer_totals
SELECT customer_id, COUNT(*), SUM(amount) FROM orders GROUP BY customer_id;

-- in another session
SET 'execution.runtime-mode' = 'batch';
SELECT * FROM customer_totals WHERE customer_id = 42;  -- point lookup in RocksDB
SELECT SUM(revenue) FROM customer_totals;               -- lake snapshot + newer rows in Fluss
SELECT COUNT(*) FROM customer_totals$lake;             -- lake copy only

Fluss, Kafka and Paimon#

Fluss sits between an event log and a lake table. Rows are summarised from Fluss’s own comparison page and from Jack Vanlightly’s reading of the code (he works at Confluent, which runs Flink but not Fluss).

	Apache Kafka	Apache Fluss	Apache Paimon
What it stores	Ordered logs of records; the payload format is up to producers	Log tables and primary-key tables with an enforced schema	Append and primary-key tables as files in object storage
Reading part of the data	Consumers fetch whole records	The server prunes columns and skips batches by filter	Engines read only the needed columns of the data files
Updates and lookups	Appends only; state for joins and deduplication lives in the consumer	Upserts, partial updates, deletes and key lookups; the server writes the changelog	Upserts merged by writers and compaction; changelogs cost lookups or full compactions
Freshness	Seconds	Sub-second	Minutes
Older data	Tiered storage, but historical reads still pass through the brokers	Clients read tiered segments directly, and tables are copied into a lake format	Is the long-term store
Who can use it	Clients in almost every language	Flink and Spark connectors; Java, Rust, Python and C++ clients	Flink, Spark, Trino and other lake engines

Three places a Fluss table’s data lives#

TabletServer disks

hot

The newest log segments, replicated to follower servers, and the RocksDB state of primary-key tables, which isn’t tiered and must fit on disk. table.log.local-ttl limits how much log stays local.

Remote storage

S3, HDFS, OSS

Older log segments and periodic RocksDB snapshots, uploaded by the TabletServers. Readers catching up download segments straight from storage, and a new bucket leader restores state from the latest snapshot.

Lake tables

Paimon, Iceberg, Hudi, Lance

Written by the tiering service, a Flink job, every table.datalake.freshness interval (3 minutes by default). Other engines query them as ordinary lake tables, and the lake’s own retention rules apply.

See it in a real stack

No architecture uses Apache Fluss yet. Drop it into the playground and wire it up yourself.Open playground →

Worth exploring

Article· September 2025Understanding Apache FlussJack VanlightlyA long, detailed walk through Fluss internals, reverse-engineered from the code: log tables, primary-key tables that emit changelogs, and tiering to lakehouse tables. The author works at Confluent, which runs Flink but not Fluss, and says so up front.

All resources →

Where it came from2023–2026

At Alibaba most real-time warehouses were built from Flink jobs reading Kafka, and the Flink team saw the same problems again and again: Kafka can’t update a record, so jobs keep huge state to deduplicate; it can’t be queried for debugging; backfills from tiered storage pass through the brokers; and every consumer reads every column, although Alibaba’s Flink SQL jobs used only 49% of upstream columns on average. Lakehouse formats such as Paimon, also born in the Flink community, fixed updates and history but were file-based, so minutes were their practical floor.

In July 2023 Jark Wu and the Flink team at Alibaba Cloud started Fluss, short for “Flink Unified Streaming Storage” and also German for river. After more than a year inside Alibaba it was announced at Flink Forward Berlin in 2024 and open-sourced under Apache 2.0 on 29 November 2024 in Shanghai. Alibaba donated it to the Apache Software Foundation, where it entered the Incubator on 5 June 2025.

It graduated to a top-level project on 16 July 2026, with 157 contributors, and 1.0 in September 2026 recast it as a real-time data layer for AI applications as well as Flink.

Why it’s built this way

Tables with schemas instead of byte streams: Knowing the columns lets the server store Arrow batches and return only the columns and rows a query needs, which cuts the network traffic Kafka spends sending whole records.
Kafka’s replication protocol, adapted: Rather than invent a new log, Fluss copied and adapted Kafka’s proven leader-follower replication for each bucket, then added the table layer on top.
RocksDB behind primary-key tables: Reading the current row before each write lets the server produce an exact changelog cheaply, something Paimon could only do with lookups or full compactions, and makes the table usable for lookup joins.
Hand old data to open lake formats: Instead of building its own long-term format, Fluss tiers data into Paimon and later Iceberg, Hudi and Lance, so history is stored once in a form every lake engine can read.

Timeline

2023
Alibaba Cloud’s Flink team starts Fluss in July
2024
Announced at Flink Forward Berlin; open-sourced on 29 November
2025
Enters the Apache Incubator on 5 June
2025
0.8.0 in November, the first Apache release
2026
Graduates to a top-level Apache project on 16 July
2026
Fluss 1.0 in September adds an HTTP gateway and Hudi tiering

Sources29 · facts checked 6 October 2026

Improve this page on GitHub