Cloud Parts

DenormalizedArchived

An embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.

What it is
Stream processing
Type
Open source
Made by
Denormalized (Probably Nothing Labs, Y Combinator Summer 2022), now inactive
License
Apache 2.0 (from the first commit in July 2024)
Status
Archived since 18 December 2024
Facts checked
6 October 2026
Website Source code (archived)

What it is

Denormalized ran inside your own program instead of as a cluster. A Rust or Python program created a Context, pointed it at a Kafka topic and gave it one sample JSON event, from which it inferred the schema, plus the name of the event-time column. On that stream you chained DataFrame-style steps built from DataFusion expressions: filter, select, with_column, stream joins, and window, which grouped events by key into tumbling or sliding windows and aggregated them. Kafka messages were decoded into Apache Arrow record batches and run through streaming operators built on a fork of Apache DataFusion. Results went to another Kafka topic, to the console, to a Python function that received each Arrow batch, or into a Feast feature store through a helper class. With checkpointing switched on, operator state was saved in SlateDB, an embedded key-value store, at a path you chose, and a restarted job recovered from it.

It was built by a two-person San Francisco startup and published under Apache 2.0 for free; the plan was to find design partners, and the website promised millions of events per second on a single node. It never reached a stable release. The Python package stopped at 0.0.13 in December 2024, the repository has had no commits since, and Y Combinator lists the company as inactive. pip install denormalized still works, but against a pinned DataFusion fork and Arrow 53, with nobody fixing bugs.

Why teams pick it

  • Embeddable: no cluster, JVM or separate service, just a library inside a Rust or Python process
  • Built on DataFusion and Arrow, so expressions, functions and record batches are the same ones the DataFusion ecosystem uses
  • Infers a topic’s schema from one sample JSON event, so a first pipeline is a few lines
  • Small, readable Apache 2.0 code base if you want to study or fork a streaming engine on DataFusion

Watch out for

  • Abandoned: no commits or releases since December 2024, and the company behind it is inactive
  • Never got past 0.0.x; its own README calls it work in progress
  • Kafka was the only real-time source and sink, and there was no SQL interface; issues asking for SQL and NATS support are still open
  • Depends on a fork of DataFusion pinned to one commit rather than a release, so moving to a newer DataFusion means redoing that fork
  • One process only, with opt-in checkpointing on an early SlateDB version (0.2)

Alternatives, recommended first

What all stream processing have in common →
ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

Per-sensor stats every second from one Kafka topic to another (last release, 0.0.13) · python

# pip install denormalized==0.0.13
import json

from denormalized import Context
from denormalized.datafusion import col, lit
from denormalized.datafusion import functions as f

sample_event = {"occurred_at_ms": 100, "sensor_name": "foo", "reading": 0.0}

ds = Context().from_topic(
    "temperature",             # Kafka topic to read
    json.dumps(sample_event),  # the schema is inferred from this sample
    "localhost:9092",
    "occurred_at_ms",          # event-time column
)

(
    ds.window(
        [col("sensor_name")],
        [
            f.count(col("reading"), distinct=False, filter=None).alias("count"),
            f.max(col("reading")).alias("max"),
            f.avg(col("reading")).alias("average"),
        ],
        1000,  # window length in milliseconds
        None,  # no slide: tumbling windows
    )
    .filter(col("max") > lit(113))
    .sink_kafka("localhost:9092", "hot-sensors")
)

Where to go instead#

Maintained projects that cover what Denormalized set out to do.

Arroyo

Rust, DataFusion

The same foundations (Rust, Arrow, DataFusion) as a stream processing engine with SQL pipelines, windows, joins and checkpoints to object storage. It runs as a service or cluster rather than inside your program; MIT or Apache 2.0, owned by Cloudflare.

Quix Streams

Python library

Closest to Denormalized’s Python API: a library you import into your own program, with a pandas-like streaming DataFrame over Kafka and per-key state in RocksDB backed up to Kafka. Apache 2.0.

RisingWave

SQL database

If you would rather write SQL and query the results: a streaming database that keeps materialized views up to date and speaks the PostgreSQL protocol. A cluster, not a library; Apache 2.0 core.

See it in a real stack

No architecture uses Denormalized yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2022–2024

Amey Chaugule, who had spent four years on Uber’s marketplace dynamics team, and Matt Green went through Y Combinator’s Summer 2022 batch as Probably Nothing Labs, building Gigabrain, a blockchain indexer that relied on stateful stream processing. The company later pivoted to Denormalized, a general-purpose stream processing engine.

Their argument, in an August 2024 Show HN post, was that DuckDB had shown how far a single machine can go, and that streaming jobs rarely compute over more than tens of gigabytes at a time, yet Flink makes you start a distributed JVM cluster for even the simplest stream. Denormalized was meant to be embedded in an application and handle hundreds of thousands of events per second, with a Flink-like dataflow API on top of Apache DataFusion and Arrow. Python bindings arrived on PyPI in October 2024, followed by checkpointing in SlateDB and an integration with the Feast feature store.

Releases stopped with 0.0.13 in December 2024. Open issues asked for SQL and more sources, but no further code was pushed, and Y Combinator now lists the company as inactive.

Why it’s built this way

An embeddable library instead of a cluster
The founders argued that most streaming workloads fit on one machine, as DuckDB had shown for analytics, so a JVM cluster was overhead most users didn’t need.
Build on DataFusion and Arrow
Reusing an existing columnar query engine gave them expressions, functions, joins and aggregation without writing them from scratch; they kept a fork to add streaming operators.
Infer the schema from a sample event
Their pitch was getting from an idea to a running pipeline quickly; deriving the schema from one JSON example saved writing it by hand.
Rust core with Python bindings
The engine stayed in Rust, and Python bindings, promised in the Show HN post, opened it to Python data pipelines and to the Feast feature store.

Timeline

  1. 2022

    Founders join Y Combinator’s Summer 2022 batch as Probably Nothing Labs

  2. 2024

    Denormalized repository opens in July under Apache 2.0; Show HN in August

  3. 2024

    Python bindings on PyPI from October; Feast integration and SlateDB checkpoints follow

  4. 2024

    Last release, 0.0.13, on 17 December; last commit the next day

Sources11 · facts checked 6 October 2026

Improve this page on GitHub