Pathway

A Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.

What it is: Stream processing
Type: Commercial product
Made by: Pathway Technology Inc. (with its French affiliate NavAlgo SAS)
License: Business Source License 1.1 (source-available, not OSI open source) since the first public release in July 2023; each version becomes Apache 2.0 four years after merge, and no earlier than July 2027. Scale features need a licence key; Enterprise is under a separate commercial licence
Facts checked: 6 October 2026

What it is

A Pathway program is an ordinary Python script. Input connectors such as pw.io.kafka.read or pw.io.s3.read turn sources into tables with a schema, and operations like filter, join, groupby().reduce() and time windows describe how to derive new tables. None of this runs anything: it only builds a dataflow graph. When the script calls pw.run(), a Rust engine built on Pathway’s own forks of Timely and Differential Dataflow takes over and runs until the process is stopped. Every change to an input is an insertion or a deletion, and only the parts of the graph that change are recomputed: a running sum is adjusted, not re-added. Output connectors send results on as a stream of changes, applied as upserts in PostgreSQL or appended with diff and time columns to a file. Tables and operator state live in memory; pathway spawn --threads 4 runs the same script on several worker threads, and the same code processes a fixed data set in static (batch) mode. For restarts, Pathway can save snapshots of its state and the input offsets to local disk or S3, which gives at-least-once recovery after a crash.

Pathway is source-available under the Business Source License, not open source. The free Community edition may run in production on one machine, up to 8 GB of RAM and 4 CPU cores per node, by anyone except companies offering it as a hosted service to others. A licence key set with pw.set_license_key unlocks the Scale tier: 16 GB per node on the free key, larger paid tiers, and extra connectors (Iceberg, Delta Lake, Kinesis, SharePoint, PostgreSQL change capture), full persistence and monitoring. Enterprise, under a separate commercial licence, adds multi-node scaling, hot failover and exactly-once consistency. Much of the marketing is about AI: the LLM extension pack parses, splits and embeds documents into a live vector index, so a RAG application’s answers follow changes in sources such as Google Drive or SharePoint.

Why teams pick it

Python all the way: pip install pathway, type-hinted schemas, any Python library in user-defined functions, and notebooks for development
Built-in operators run in Rust on several threads, outside Python’s global interpreter lock
The same code runs as a batch job over historical data or as a live stream, and late events update earlier results instead of being dropped
Ready-made LLM tooling: document parsers, splitters, embedders, a live vector and hybrid index, RAG question answerers, and LangChain and LlamaIndex integrations
Many connectors in the free tier, including Kafka, S3, files, Google Drive and PostgreSQL output, plus an Airbyte connector for hundreds more sources

Watch out for

Not open source: free production use stops at one machine with 8 GB of RAM and 4 cores per node; past that you need a key or a commercial licence, and hosting it for others is excluded
Iceberg, Delta Lake, Kinesis, SharePoint and PostgreSQL change-capture connectors, full persistence and monitoring need a Scale key, and using one sends anonymized usage data to Pathway
Recovery after a crash is at-least-once, so outputs can repeat; exactly-once consistency is an Enterprise feature
State is held in memory, and a window keeps every row it has seen unless you set a cutoff for late data
Linux and macOS only (Windows through a VM), and without an Enterprise licence new instances can’t be launched automatically

Alternatives

What all stream processing have in common →

Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.

ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.

BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.

Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.

DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.

FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.

Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.

ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.

Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.

Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.

RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.

StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

What Pathway uses

Flag IP addresses with five or more failed logins in a minute, from one Kafka topic to another · python

import pathway as pw

rdkafka_settings = {
    "bootstrap.servers": "localhost:9092",
    "group.id": "login-monitor",
    "auto.offset.reset": "earliest",
}

class Login(pw.Schema):
    username: str
    ip_address: str
    successful: bool
    time: int  # Unix seconds: when the attempt happened

logins = pw.io.kafka.read(rdkafka_settings, topic="logins", format="json", schema=Login)
failed = logins.filter(~pw.this.successful)

# one-minute windows by event time, per IP; a window stops changing 30 s after it ends
per_minute = failed.windowby(
    failed.time,
    window=pw.temporal.tumbling(duration=60),
    instance=pw.this.ip_address,
    behavior=pw.temporal.common_behavior(cutoff=30),
).reduce(
    ip_address=pw.this._pw_instance,
    window_end=pw.this._pw_window_end,
    failures=pw.reducers.count(),
)

suspicious = per_minute.filter(pw.this.failures >= 5)
pw.io.kafka.write(suspicious, rdkafka_settings, topic_name="suspicious-ips", format="json")

pw.run()  # nothing runs before this; then the Rust engine processes updates until stopped
# on 4 worker threads:  pathway spawn --threads 4 python monitor.py

Community, Scale or Enterprise#

One package, three sets of rights. Limits and features as listed by Pathway on 2026-10-06; it publishes no prices.

	Community	Scale	Enterprise
Licence	BSL 1.1	BSL 1.1, activated with a licence key	Separate commercial licence
Resources	8 GB RAM and 4 CPU cores per node	16 GB RAM and 4 cores per node on the free key; larger paid tiers	Listed up to 24 TB RAM and 128 cores across 40 nodes
What it adds	Core transformations, most connectors (Kafka, S3, files, Google Drive, PostgreSQL output), LLM extension pack	Iceberg, Delta Lake, Kinesis, SharePoint and PostgreSQL change-capture connectors; full persistence; monitoring	Horizontal scaling across nodes, hot failover, Helm chart, exactly-once consistency, managed hosting
Conditions	One machine per installation; not for providers offering it as a service to third parties	Anonymized usage data sent to Pathway through OpenTelemetry; scaling up or down must be done by hand	Negotiated with Pathway

Under the BSL, each version of the code becomes Apache 2.0 four years after it was merged, and in no case before July 20, 2027.

See it in a real stack

No architecture uses Pathway yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2020–2026

Pathway was founded in Paris by a team led by Zuzanna Stamirowska, a complexity scientist, with Jan Chorowski, a machine learning researcher, as CTO and Adrian Kosowski, a computer scientist, as chief scientific officer. They had been building the framework since 2020 for a specific problem: streaming data keeps changing, and developers had to handle those updates by hand, keep separate code for real-time and historical data, and debug against data they couldn’t see. Their answer was to let people write pipelines in Python as if the data were static, and let the engine propagate the changes.

Pathway opened a beta to all developers in December 2022, together with a $4.5 million funding round, and published its source on GitHub with release 0.2.0 in July 2023, under the Business Source License. The engine was built on forked copies of Frank McSherry’s Timely and Differential Dataflow. From January 2024 the framework added an LLM extension pack, and Pathway pitched it for live RAG pipelines. A May 2024 licence update put enterprise features in the same source tree behind a licence key, and in December 2024 the company raised a $10 million seed round to grow in the United States; it is now headquartered in Palo Alto.

In October 2025 Pathway introduced BDH, a “post-Transformer” AI architecture, and its website now presents the company as an AI lab. The framework was renamed Pathway Live Data Framework in 2026 and is still released regularly.

Why it’s built this way

Python on top, Rust underneath: Python is what data scientists and ML engineers already use, but it is slow and its global interpreter lock limits parallelism. Python only describes the typed pipeline; the Rust engine stores the tables and runs most operators.
Building on Differential Dataflow: Differential Dataflow already maintained results incrementally under insertions and deletions, including loops. Forking it gave Pathway consistent updates and iteration without writing an incremental engine from scratch.
Updates instead of append-only streams: Treating every change as an insertion or deletion lets late or corrected data revise earlier results, so the same code gives the same answer on a live stream and on a batch replay.
Business Source License with a licence key: Pathway wanted the core free for production use while keeping cloud providers from reselling it, and charging for scale and enterprise features. The licence key enforces resource limits in the code itself.

Timeline

2020
Work on the framework starts at the Paris-founded start-up
2022
Open beta and a $4.5 million funding round in December
2023
Source published on GitHub under BSL 1.1 with release 0.2.0 in July
2024
LLM extension pack in January; licence keys in May; $10 million seed in December
2025
Pathway introduces its BDH AI architecture
2026
Renamed Pathway Live Data Framework; 0.33.0 released in September

Sources21 · facts checked 6 October 2026

Improve this page on GitHub