Cloud Parts

Bytewax

A Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.

What it is
Stream processing
Type
Open source
Made by
Community-maintained since 2025; built by Bytewax, Inc., which is no longer operating
License
Apache 2.0
Facts checked
6 October 2026
Website Source code

What it is

Bytewax is a Python library with a Rust core. You describe a Dataflow of operators (map, filter, stateful_map, fold_window, join) and give them plain Python functions. python -m bytewax.run compiles that graph into Timely Dataflow, a Rust dataflow runtime that Bytewax embeds through PyO3, and every worker calls your Python code for each item. Workers are threads; you can run several in one process (-w) or start processes on several machines (-i, -a), and items that must move to another worker for a keyed step are pickled and sent over the network. Inputs stamp items with epochs, short wall-clock intervals. At each snapshot interval every stateful operator pickles its state, and internal recovery operators write those snapshots and the input positions (including Kafka offsets, which Bytewax keeps itself rather than in consumer groups) into recovery partitions, which are SQLite files. After a crash, the next run finds the newest epoch that every partition finished, rewinds state and inputs to it, and can do so with a different number of workers. Output to Kafka is at-least-once: messages from that epoch are written again.

The code is Apache 2.0. Bytewax, Inc., founded by Zander Matheson, open-sourced it in 2022 and tried to sell a paid Bytewax Platform for running dataflows on Kubernetes. In May 2025 the company was no longer commercially viable and the core team stopped maintaining the project. A year later Matheson marked the repository as community-maintained, with himself as the only listed maintainer. Commits since then have updated CI and dependencies (including PyO3 for Python 3.13 and 3.14), but as of October 2026 there has been no release since 0.21.1 on 25 November 2024, whose wheels cover Python 3.8 to 3.12. The bytewax.io website and hosted docs no longer resolve; the docs source lives in the repository.

Why teams pick it

  • Python all the way: operators take ordinary Python functions, so pandas, NumPy or online-learning libraries run inside the dataflow, installed with pip and no JVM
  • Stateful operators with recovery: stateful_map, joins, and tumbling, sliding and session windows on event or processing time
  • Rescaling between runs: on resume, saved state follows its keys to however many workers the new run has
  • The same script runs on a laptop, as several processes, or on Kubernetes with the published Helm chart

Watch out for

  • No company behind it and one volunteer maintainer: the last release is 0.21.1 (November 2024), and there are no published wheels for Python 3.13 or newer
  • The paid Platform is gone, the website and hosted docs are offline, and the waxctl downloads repository is archived
  • Recovery partitions are SQLite files you create up front, in a count you can’t change later, and back up yourself, for example with cron or Litestream
  • Kafka output is at-least-once, so sinks see duplicates after a resume and should be idempotent; a resume can also replay up to one snapshot interval
  • Epochs follow each worker’s wall clock, so a cluster assumes roughly synchronized clocks

Alternatives

What all stream processing have in common →
Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

Average temperature per sensor per minute, from one Kafka topic to another (Bytewax 0.21) · python

# dataflow.py
import json
from datetime import datetime, timedelta, timezone

import bytewax.operators as op
import bytewax.operators.windowing as win
from bytewax.connectors.kafka import KafkaSinkMessage
from bytewax.connectors.kafka import operators as kop
from bytewax.dataflow import Dataflow
from bytewax.operators.windowing import EventClock, TumblingWindower

BROKERS = ["kafka-1:9092"]
flow = Dataflow("sensor-averages")

msgs = kop.input("in", flow, brokers=BROKERS, topics=["sensor-readings"])
readings = op.map("parse", msgs.oks, lambda m: json.loads(m.value))
by_sensor = op.key_on("key", readings, lambda r: r["sensor_id"])

clock = EventClock(
    lambda r: datetime.fromisoformat(r["ts"]),  # e.g. "2026-10-06T12:00:03+00:00"
    wait_for_system_duration=timedelta(seconds=10),  # wait for late readings
)
windower = TumblingWindower(
    length=timedelta(minutes=1),
    align_to=datetime(2026, 1, 1, tzinfo=timezone.utc),
)
sums = win.fold_window(
    "sum", by_sensor, clock, windower,
    builder=lambda: (0.0, 0),
    folder=lambda acc, r: (acc[0] + r["temp_c"], acc[1] + 1),
    merger=lambda a, b: (a[0] + b[0], a[1] + b[1]),
)

def encode(item):
    sensor_id, (window_id, (total, n)) = item
    body = {"sensor_id": sensor_id, "window": window_id, "avg_c": total / n}
    return KafkaSinkMessage(sensor_id.encode(), json.dumps(body).encode())

out = op.map("encode", sums.down, encode)
kop.output("out", out, brokers=BROKERS, topic="sensor-averages-1m")

# once:  python -m bytewax.recovery recovery/ 4
# run:   python -m bytewax.run dataflow:flow -w 2 -r recovery/ -s 30 -b 0

Project status, October 2026#

What a new user would find today, checked 2026-10-06.

ItemStatus
Latest release0.21.1, 25 November 2024 (GitHub and PyPI)
Python versions with wheels3.8 to 3.12; 3.13 and 3.14 support is merged but unreleased
CompanyBytewax, Inc. said in May 2025 it was not commercially viable; the core team stepped back
MaintainersFounder Zander Matheson, best effort, looking for more maintainers in issue #560
Last commit to main20 June 2026 (CI and dependency upgrades); several pull requests open
Website, docs, `waxctl`bytewax.io doesn’t resolve; docs source is in the repo; waxctl-downloads is archived
RepositoryNot archived; Apache 2.0, so forks are allowed (one user publishes bytewax-fork on PyPI)

Where Bytewax users are looking#

Alternatives named by users in the project’s own “State of the project?” issue.

Quix Streams

Apache 2.0

Pure-Python library on Kafka with RocksDB state and changelog topics, released every few weeks. Needs Kafka for everything, but no cluster of its own.

Pathway

BSL 1.1

Python API on a Rust engine based on Differential Dataflow, a close cousin of Timely. Free production use is limited to one machine; the licence is source-available, not open source.

Arroyo

MIT or Apache 2.0

A Rust stream engine where pipelines are SQL, with checkpoints to object storage, developed by the Arroyo team at Cloudflare.

Stay on 0.21 or a fork

your own risk

Existing dataflows keep running on Python 3.12 or older. The repo accepts pull requests, and a user-run fork, bytewax-fork, carries extra fixes.

See it in a real stack

No architecture uses Bytewax yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2021–2026

In the early 2020s the main stateful stream processors, such as Flink, Spark and Kafka Streams, ran on the JVM, and most meant running a cluster. Zander Matheson, who had worked on data at Heroku and GitHub, saw teams spend months on infrastructure before they could build fraud detection or monitoring, while the people writing that logic worked in Python.

His company, Bytewax, open-sourced its framework on 7 February 2022: a Python library on top of Timely Dataflow, a Rust dataflow engine, so it could scale from one machine to a cluster without a JVM. Version 0.17 in 2023 rebuilt recovery around SQLite partitions so dataflows could be rescaled, and 0.18 that December reshaped the API into stand-alone operator functions. Microsoft’s venture fund M12 invested in 2023 to support the open source and a paid Bytewax Platform.

The platform didn’t find enough customers. In May 2025 Matheson told users the company was not viable and the core team would stop maintaining the project. A year later he wrote on GitHub that Bytewax “wasn’t commercially viable and the original core team isn’t maintaining the project full-time anymore”, and set it up for community maintenance.

Why it’s built this way

A Python API on a Rust engine
The team wanted Python’s ecosystem for the logic and a fast, scalable back end, without making users learn another language or run another distributed system. Timely Dataflow supplied the Rust runtime and networking.
Epochs from Timely’s progress tracking
Timely tells every operator when an epoch is finished across the whole cluster. Bytewax uses that signal both for backpressure and to send a consistent wave of snapshots through every stateful operator.
Its own recovery store instead of Kafka consumer groups
To rewind every input and every operator to the same snapshot, and to change the worker count on resume, Bytewax keeps offsets and state together in SQLite recovery partitions. Version 0.17 dropped the earlier Kafka-based recovery for this design.

Timeline

  1. 2021

    Zander Matheson starts Bytewax, Inc.

  2. 2022

    Open-sourced on 7 February as a Python framework on Timely Dataflow

  3. 2023

    0.17 reworks recovery and allows rescaling; M12 invests

  4. 2024

    0.21.1 in November, the last release so far

  5. 2025

    Company no longer viable in May; core team stops maintaining

  6. 2026

    Repository marked community-maintained in May

Sources26 · facts checked 6 October 2026

Improve this page on GitHub