Cloud Parts

Cloudflare Basin PipelinesRenamed

Cloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.

What it is
Stream processing
Used for
ETL & ELT pipelines
Type
Commercial product
Made by
Cloudflare, Inc.
License
Proprietary
Status
Renamed. Called Cloudflare Pipelines until 1 October 2026, when it became generally available as Basin Pipelines; existing resources and configurations keep working.
Facts checked
12 October 2026
Website

What it is

A setup has three parts. A stream is a durable, buffered queue that accepts JSON events from an HTTP endpoint, a Worker binding (env.STREAM.send(records)) or Cloudflare Logpush, and can validate them against a schema. A pipeline is a SQL statement, INSERT INTO sink SELECT … FROM stream, that filters, reshapes, casts or hashes each event on the way through. A sink writes the result with exactly-once delivery, either as Iceberg tables in Basin Catalog or as JSON or Parquet files in an R2 bucket, rolling a new file by size (5 MB or more) or time (every 10 seconds or more). One stream can feed several pipelines. The engine underneath comes from Arroyo, the Rust stream processor Cloudflare bought in 2025.

Transforms are stateless for now: no windows, joins or aggregations, which Cloudflare lists as planned. An account can have 20 streams, 20 pipelines and 20 sinks, a request can carry 5 MB, and the limits page allows 1 GB/s per stream. It needs the Workers Paid plan. Ingesting into a stream is free; SQL transforms cost $0.04 per GB beyond 50 GB a month, and sinks $0.03 per GB for JSON or $0.06 per GB for Parquet and Iceberg beyond 50 GB, measured on uncompressed data, plus normal R2 and Basin Catalog charges. In Cloudflare’s example, 500 GB a month filtered down to 300 GB of Iceberg costs $33.

Why teams pick it

  • No Kafka, Flink or Spark to run: events go from a Worker or HTTP call to a queryable Iceberg table with one SQL statement
  • Exactly-once delivery into Iceberg, Parquet or JSON on R2, with schema validation and typed Worker bindings that catch mismatches before deploy
  • Free ingestion and per-GB pricing for transforms and sinks, with no hourly charges
  • Cloudflare Logpush can feed a stream, so Cloudflare’s own logs can be filtered and stored as Iceberg tables

Watch out for

  • Stateless SQL only: no windowed aggregations, joins or materialized views yet
  • Low default quotas (20 streams, pipelines and sinks per account) and 5 MB per ingestion request
  • Writes only to R2 and Basin Catalog; streams accept JSON events only
  • Pipelines created before 25 September 2025 are on a deprecation path, and the October 2026 rename means older wrangler pipelines guides are out of date

Alternatives

What all stream processing have in common →
Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

Create a stream, a SQL pipeline and an Iceberg sink, then send events · bash

# interactive: creates the stream (with schema), the Basin Catalog sink and the SQL
npx wrangler basin pipelines setup --name ecommerce

# the pipeline is one SQL statement, for example:
#   INSERT INTO ecommerce_sink
#   SELECT user_id, event_type, product_id, amount
#   FROM ecommerce_stream WHERE event_type = 'purchase';

curl -X POST "https://$STREAM_ID.ingest.cloudflare.com" \
  -H "Content-Type: application/json" \
  -d '[{"user_id": "user_12345", "event_type": "purchase", "product_id": "widget-001", "amount": 29.99}]'

# from a Worker instead: "pipelines": [{ "binding": "STREAM", "stream": "<STREAM_ID>" }]
#   await env.STREAM.send([{ user_id: "user_12345", event_type: "purchase" }]);

See it in a real stack

No architecture uses Cloudflare Basin Pipelines yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2023–2026

Arroyo was started in 2023 by Micah Wylde, who had worked on stream processing at Lyft and Splunk, to make real-time pipelines usable by people who aren’t streaming experts. Apache Flink had the power but, in Micah Wylde’s words, an API too hard and low-level for most users and stateful services that needed endless operations. Arroyo answered with SQL as the interface, Rust instead of the JVM, and state kept in object storage, and was open-sourced in the summer of 2023.

On 10 April 2025 Cloudflare put Pipelines into open beta, at first a way to batch JSON events into R2, and announced it had acquired Arroyo to bring SQL transforms and Iceberg output to it. On 25 September 2025 it relaunched Pipelines on the new architecture, with streams, SQL pipelines and sinks into R2 Data Catalog, as part of the Cloudflare Data Platform alongside R2 SQL.

On 1 October 2026 the platform became generally available as Cloudflare Basin, and Pipelines became Basin Pipelines, by then with tens of thousands of pipelines created, Logpush as a source, typed bindings and Terraform support.

Why it’s built this way

SQL as the only transformation language
Arroyo’s founders chose SQL because it is easy to use, widely known and declarative, and Cloudflare kept it so that validation and reshaping happen at ingestion, “shifting left” before data lands in storage.
Separate streams, pipelines and sinks
The September 2025 architecture made the buffer, the transform and the destination distinct resources, so one stream can be read by several pipelines that route the same data to different tables or files.
Stateless transforms first
Filtering and reshaping cover most ingestion; Cloudflare said it would bring Arroyo’s stateful features (aggregations, joins, materialized views) later and may price them separately.

Timeline

  1. 2023

    Arroyo, the engine it is built on, is open-sourced

  2. 2025

    Pipelines open beta and Arroyo acquisition on 10 April

  3. 2025

    Rebuilt with SQL transforms and Iceberg sinks on 25 September

  4. 2026

    Generally available and renamed Basin Pipelines on 1 October

Sources10 · facts checked 12 October 2026

Improve this page on GitHub