Cloud Parts

StreamFusion

An open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.

What it is
Stream processing
Type
Open source
Made by
Jordan Epstein and contributors, in the datafusion-contrib GitHub organization (not an Apache project)
License
Apache 2.0 (LICENSE file since the repository opened in datafusion-contrib, July 2026)
Facts checked
6 October 2026
Website Source code

What it is

StreamFusion is not a separate engine. You put its loader and core JARs in the lib directory of a Flink 2.2 cluster and submit ordinary Flink SQL jobs, unchanged. After Flink’s planner has optimized a query, StreamFusion inspects the plan: if every operator between the source and the sink has a native version that gives byte-identical results, it swaps them for Rust operators called over JNI; if even one doesn’t, the whole query runs on stock Flink. The native operators pass Apache Arrow column batches to each other and use Apache DataFusion for expressions, hash joins and aggregates, with streaming operators ported from Arroyo, so rows are converted to columns only at the edges. For Kafka, Flink’s own client still handles partitions, offsets and transactions while Rust decodes and encodes the JSON, CSV, Avro, Protobuf or CDC bytes straight to and from Arrow. Flink also keeps checkpoints, rescaling and monitoring: checkpoint barriers flow through the JVM as usual and trigger each native operator’s snapshot, stored as opaque keyed state that Flink can redistribute. State sits in memory or in a RocksDB instance owned by the Rust code, with incremental checkpoints.

Native coverage includes TUMBLE, HOP, SESSION and CUMULATE windows, OVER aggregates, regular, interval, window, temporal and lookup joins, updating GROUP BY, Top-N and deduplication, plus Parquet, ORC and Delta Lake sinks and Paimon sources and sinks. MATCH_RECOGNIZE, lateral table functions and PyFlink UDFs stay on Flink. Jordan Epstein started it in June 2026 and has written most of it; it is free under Apache 2.0, with no company or paid edition behind it, and so far ships only release candidates (0.1.0-rc1 in August 2026 to 0.1.0-rc4 on 6 October 2026) through Maven Central.

Why teams pick it

  • Drop-in: no query rewriting and no application change, and streamfusion.native.enabled=false puts everything back on stock Flink
  • Parity first: it only substitutes what it can reproduce byte for byte, checked by a parity harness and Flink’s own SQL tests in CI, and it can log why each plan node fell back
  • Accelerates stateful SQL (windows, joins, changelog aggregation, Top-N), not just stateless filters and projections
  • Faster on the project’s own Nexmark runs: a geometric mean of 1.3 to 1.4 times Flink’s throughput with in-memory state and 2.8 to 3.4 times with RocksDB, with byte-identical output
  • Open source under Apache 2.0, where the other native Flink engines (Alibaba’s Flash, Ververica’s VERA-X) are proprietary

Watch out for

  • Pre-release: only 0.1.0 release candidates so far, internal APIs may change before 1.0, and savepoint upgrades across versions haven’t been validated
  • Pinned to Flink 2.2.0 and 2.2.1 while Flink itself is at 2.3.0; a Flink 1.18 build exists for development only, and the published native libraries cover Linux x86-64 and Apple Silicon Macs, not Linux ARM64
  • SQL and Table API only: DataStream jobs, PyFlink UDFs, MATCH_RECOGNIZE and lateral table functions stay on Flink, and one unsupported operator sends the whole query back
  • Mostly one developer’s project (over 1,300 of about 1,660 commits), written with heavy help from AI coding tools by his own account, so its future rests on a small group
  • Native state and buffers come out of the TaskManager’s off-heap memory, which you have to size, and each connector and format needs its own extra JAR

Alternatives

What all stream processing have in common →
Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

Add StreamFusion to a Flink 2.2.1 distribution and run an ordinary SQL job · bash

cd "$FLINK_HOME"   # Flink 2.2.1 on Java 17
SF=0.1.0-rc4
M=https://repo1.maven.org/maven2

# the loader and core runtime
curl -fL -o lib/00-streamfusion-loader.jar "$M/tech/streamfusion/streamfusion-loader/$SF/streamfusion-loader-$SF.jar"
curl -fL -o lib/streamfusion-core.jar "$M/tech/streamfusion/streamfusion-core/$SF/streamfusion-core-$SF-runtime.jar"

# Kafka with JSON: StreamFusion's modules plus Flink's own Kafka connector
curl -fL -o lib/streamfusion-kafka.jar "$M/tech/streamfusion/streamfusion-kafka/$SF/streamfusion-kafka-$SF.jar"
curl -fL -o lib/streamfusion-json.jar "$M/tech/streamfusion/streamfusion-json/$SF/streamfusion-json-$SF.jar"
curl -fL -o lib/flink-connector-kafka-5.0.0-2.2.jar "$M/org/apache/flink/flink-connector-kafka/5.0.0-2.2/flink-connector-kafka-5.0.0-2.2.jar"
curl -fL -o lib/kafka-clients-4.2.0.jar "$M/org/apache/kafka/kafka-clients/4.2.0/kafka-clients-4.2.0.jar"

# native operators draw on task off-heap memory; keep state in the Rust-owned RocksDB;
# log every plan node that stays on Flink, and why
cat >> conf/config.yaml <<'EOF'
taskmanager.memory.task.off-heap.size: 512m
state.backend.type: tech.streamfusion.state.RocksDBNativeStateBackendFactory
env.java.opts.all: -Dstreamfusion.logFallbackReasons=true
EOF

./bin/start-cluster.sh
# an unchanged Flink SQL job; EXPLAIN shows Native* nodes when it accelerates
./bin/sql-client.sh -f revenue_per_minute.sql

StreamFusion’s own Nexmark numbers#

StreamFusion throughput divided by stock Flink throughput, as published in the project’s README. Measured by the StreamFusion project, not independently.

In-memory stateRocksDB state
Geometric mean, mini-batch off1.31×2.75×
Geometric mean, mini-batch on1.44×3.41×
Range over 23 queries0.99× to 3.32×1.08× to 68.20×
  • Setup: an Apple M1 Max, parallelism 4, a 2-million-event JSON corpus read from a four-partition Kafka topic, results written exactly once to Kafka with one-second checkpoints. In-memory columns are one run each (2026-08-25); RocksDB columns are the best of two runs (2026-09-05). Query q6 is left out because Flink SQL can’t run it.
  • Seven queries use opt-in expression variants (approximate decimals, native datetime or regex behaviour) that can differ from Flink at documented edge cases. The biggest gains come from stateful queries on RocksDB (q9, q20), where StreamFusion batches state access.

See it in a real stack

No architecture uses StreamFusion yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2024–2026

Spark has had native accelerators for years: Databricks’ Photon and open-source projects such as Gluten (Velox), RAPIDS and DataFusion Comet keep Spark’s planner and scheduler and replace the code that touches each row with vectorized operators. Flink, the default for stream processing, had nothing comparable in the open: Alibaba’s Flash and Ververica’s VERA-X claimed five to ten times Flink’s speed, but both are paid products.

Jordan Epstein, a New York engineer who had spent four years making videos about distributed systems and had contributed to Iceberg, DataFusion, Comet, Arrow Java and Paimon, decided to build an open one. He made the first commit on 18 June 2026; in July the code moved to the datafusion-contrib GitHub organization. He modelled the project on Comet, ported streaming operators from Arroyo and RisingWave, and, working alone alongside a day job, wrote much of it with AI coding tools, relying on a parity harness against Flink to check the results.

The first release candidate came out on 15 August 2026 with a launch post claiming over 33% more throughput than Flink on Kafka-to-Kafka Nexmark runs. A handful of other contributors have joined since, and by October it had grown file and lake-table connectors (Parquet, ORC, Delta Lake, Paimon).

Why it’s built this way

Accelerate Flink instead of writing another engine
Flink’s adoption, connectors and community make it hard to leave, and Flink SQL’s limited set of operators can be reimplemented one by one, as the Spark accelerators had done.
All-or-nothing substitution with byte-identical results
Exact parity with Flink is the project’s goal. Accelerating only whole queries keeps data in Arrow between operators and avoids subtle differences; operations like regular expressions, which can behave differently in Java and Rust, stay on Flink unless you opt in.
Port operators from Arroyo and RisingWave
Both use the same Chandy-Lamport-style barrier checkpoints as Flink, so their operators fit Flink’s checkpointing; engines built on other models, such as Feldera’s DBSP, would not.
Native state as opaque keyed blobs
Flink can then move key groups between subtasks when a job rescales without understanding the bytes, provided StreamFusion reproduces Flink’s key hashing exactly in Rust.

Timeline

  1. 2024

    Alibaba presents Flash, a closed native Flink engine

  2. 2025

    Irontools announces Iron Vector (stateless) and Ververica announces VERA-X

  3. 2026

    Jordan Epstein starts StreamFusion in June; it moves to datafusion-contrib in July

  4. 2026

    First release candidate, 0.1.0-rc1, and launch post in August

  5. 2026

    Fourth release candidate, 0.1.0-rc4, on 6 October

Sources16 · facts checked 6 October 2026

Improve this page on GitHub