Arroyo

A stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.

What it is: Stream processing
Type: Open source
Made by: Cloudflare, Inc.
License: MIT or Apache 2.0, at your choice (dual-licensed since the first open-source release in 2023)
Facts checked: 5 October 2026

What it is

An Arroyo pipeline is a SQL script. CREATE TABLE statements describe the sources and sinks (Kafka topics, Kinesis streams, Parquet or JSON files in object storage, Iceberg tables), and an INSERT INTO … SELECT describes the work. The planner, built on Apache DataFusion, turns the query into a graph of operators, and workers run copies of each operator in parallel, sending batches of Apache Arrow records to each other over TCP. Operators that need memory, such as a window aggregate or a join, keep state, and Arroyo regularly writes a consistent checkpoint of that state and the source offsets to S3, GCS, Azure or R2, so a failed pipeline restarts from its last checkpoint. A WATERMARK on a timestamp column tells it when no older events are expected, which is when a window’s result is emitted.

A cluster is one binary playing several roles: a stateless API server and web UI, a controller that keeps each pipeline in its desired state, and workers started as local processes, on arroyo node machines or as Kubernetes pods. Configuration lives in SQLite on a single machine or in PostgreSQL for a production cluster. The engine is free to self-host. Cloudflare bought Arroyo in April 2025; the same team keeps developing the engine in the open and runs it as Cloudflare’s managed Pipelines service, renamed Basin Pipelines in October 2026, which today runs only stateless SQL on data headed for R2.

Why teams pick it

Pipelines are plain SQL with streaming extensions: TUMBLE, HOP and SESSION windows, stream joins, updating (CDC-style) tables, and user-defined functions in Rust or Python
Exactly-once output to Kafka (offsets kept in checkpointed state, transactional writes) and to object storage (multipart uploads tracked in state), plus two-phase commits to Iceberg
One Rust binary with no JVM: runs on a laptop with SQLite, and on Kubernetes with PostgreSQL through a Helm chart
Columnar execution on Arrow and DataFusion; the move to them in 0.10 gave three times the throughput and twenty times faster pipeline start-up, by Arroyo’s own benchmarks
A managed path exists: Basin Pipelines on Cloudflare runs the same engine, billed per GB

Watch out for

Still pre-1.0, and the last tagged GitHub release is 0.15.0 (December 2025), even though commits continue (over 130 in 2026); since the acquisition it is built by the Arroyo team inside Cloudflare
The managed Basin Pipelines service runs only stateless SQL for now; windows, joins and aggregations mean running Arroyo yourself
A production cluster needs PostgreSQL, object storage for checkpoints and a scheduler such as Kubernetes; the single-node mode is meant for testing only
A smaller connector list than Flink or Kafka Connect: PostgreSQL and MySQL are reached only through Debezium over Kafka, not directly
Session windows are capped at 24 hours

Alternatives

What all stream processing have in common →

Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.

BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.

Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.

DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.

FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.

Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.

ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.

PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.

Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.

Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.

RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.

StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

What Arroyo uses

What uses Arroyo

Revenue per customer per minute, from one Kafka topic to another, exactly once · sql

CREATE TABLE orders (
  customer_id TEXT,
  amount DOUBLE,
  created_at TIMESTAMP,
  -- events may arrive up to 10 seconds late
  WATERMARK FOR created_at AS created_at - INTERVAL '10 seconds'
) WITH (
  connector = 'kafka',
  bootstrap_servers = 'kafka-1:9092',
  topic = 'orders',
  type = 'source',
  format = 'json',
  'source.offset' = 'earliest'
);

CREATE TABLE revenue_per_minute WITH (
  connector = 'kafka',
  bootstrap_servers = 'kafka-1:9092',
  topic = 'revenue-per-minute',
  type = 'sink',
  format = 'json',
  'sink.commit_mode' = 'exactly_once'
);

-- one row per customer and minute, emitted once the watermark passes the minute
INSERT INTO revenue_per_minute
SELECT window.end AS minute_end, customer_id, orders, revenue
FROM (
  SELECT TUMBLE(INTERVAL '1 minute') AS window, customer_id,
         count(*) AS orders, sum(amount) AS revenue
  FROM orders
  GROUP BY window, customer_id
);

Self-hosted Arroyo or Basin Pipelines#

The same engine, two ways to run it. Prices checked 2026-10-05.

	Arroyo (self-hosted)	Basin Pipelines (Cloudflare)
Who runs it	You: binary, Docker image or Helm chart, plus PostgreSQL and object storage	Cloudflare; needs a Workers Paid plan
SQL	Stateful: windows, joins, updating tables, Rust and Python UDFs	Stateless transforms (filter, reshape, unnest, cast); stateful operations announced for later
Inputs	Kafka, Redpanda, Kinesis, NATS, MQTT, RabbitMQ, files, polling HTTP, WebSocket and more	Streams fed over HTTP or from Workers
Outputs	Kafka, Kinesis, Parquet or JSON files on S3, GCS, Azure or R2, Iceberg, Delta Lake, webhooks, Redis	Iceberg tables, or Parquet and JSON files, in R2
Price	Free (MIT or Apache 2.0); you pay for machines and storage	Ingest free; SQL $0.04/GB after 50 GB a month; delivery $0.03/GB (JSON) or $0.06/GB (Parquet, Iceberg) after 50 GB; plus R2 storage

Basin Pipelines bills delivery on uncompressed data, and Cloudflare says stateful operations may be priced separately when they arrive.

See it in a real stack

No architecture uses Arroyo yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2022–2026

Micah Wylde led the streaming platform teams at Lyft and Splunk, where they built two real-time data platforms on Apache Flink. Flink worked, but he found it too complex for the people who wanted streaming results, such as data scientists and product engineers, and keeping many user-written pipelines running reliably was hard.

In 2022 he and Jackson Newhouse founded Arroyo, which joined Y Combinator’s Winter 2023 batch. They launched a hosted service first and open-sourced the engine in April 2023: written in Rust, with SQL as the main interface. Early versions turned each query into Rust code and compiled it, which needed a compiler service and minutes-long cold builds. Arroyo 0.10, in 2024, replaced that with Apache Arrow and DataFusion, so one prebuilt binary could run any query.

In April 2025 Cloudflare acquired Arroyo. The engine stayed open source and self-hostable, and the team built it into Cloudflare Pipelines, which loads event streams into R2. Cloudflare made that service generally available as Basin Pipelines in October 2026.

Why it’s built this way

Rust instead of the JVM: Wylde argued that stream processors are hard cases for garbage collectors, because they create many objects quickly and keep some for unpredictable lengths of time. Rust’s borrow checker manages memory at compile time, with no collector pauses.
SQL as the main interface, for non-experts: The Flink pipelines he had supported were written by people who weren’t streaming specialists. Keeping the user-facing SQL separate from the engine’s internals lets them build real-time pipelines without learning those internals.
Arrow and DataFusion instead of generating Rust code: Compiling every pipeline needed a Rust compiler and a build service that was costly to run and slow to start. Interpreting queries on Arrow batches with DataFusion removed both and, by Arroyo’s measurements, tripled throughput.

Timeline

2022
Micah Wylde and Jackson Newhouse found Arroyo
2023
Y Combinator Winter 2023 batch; engine open-sourced in April
2024
Arroyo 0.10 rebuilds the engine on Arrow and DataFusion
2025
Cloudflare acquires Arroyo in April; its engine goes into Cloudflare Pipelines
2026
Cloudflare Pipelines becomes Basin Pipelines and is generally available

Sources25 · facts checked 5 October 2026

Improve this page on GitHub