Cloud Parts

Redpanda ConnectRenamed

Formerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.

What it is
Stream processing
Type
Open source
Made by
Redpanda Data (acquired Benthos in May 2024)
License
Mixed: the core engine (redpanda-data/benthos) is MIT, most connectors are Apache 2.0, and enterprise connectors and features are under the source-available Redpanda Community License and need a paid licence key; everything was MIT up to Benthos 4.27, and the split came with 4.28 in May 2024
Status
Renamed. Called Benthos until Redpanda bought it in May 2024.
Facts checked
5 October 2026
Website Source code

What it is

A Redpanda Connect pipeline is one YAML file with an input (a Kafka topic, an SQS queue, an HTTP endpoint, a database change stream), a list of processors that filter, enrich or reshape each message, and an output (another topic, an S3 bucket, a database, an Iceberg table). Most reshaping is written in Bloblang, a small mapping language made for it. The binary keeps no queue on disk: it acknowledges a message to the source only after the output has confirmed it, so with at-least-once sources and sinks a crash means messages are re-read, not lost. Because instances hold no state, you scale by running more copies, and streams mode runs many isolated pipelines in one process, managed through a REST API.

Ashley Jeffs built it as Benthos, under the MIT licence. Redpanda bought it in May 2024 and renamed it. The core engine now lives in its own MIT-licensed repository, the connectors moved to Apache 2.0, and a growing set of enterprise components, including every change-data-capture input, the Iceberg and Snowflake outputs and remote secrets management, are under the Redpanda Community License and stop working without a paid Enterprise licence key (a 30-day trial key is free). It runs standalone, through rpk connect, or as managed pipelines in Redpanda Cloud. WarpStream maintains Bento, an MIT fork of the code from before the licence change.

Why teams pick it

  • A whole pipeline fits in one YAML file and one static binary or container, with health checks, Prometheus-style metrics and OpenTelemetry traces built in
  • At-least-once delivery without disks or checkpoints: messages are acknowledged upstream only after delivery, so instances are stateless and easy to scale
  • A very large connector catalog: message brokers, AWS, Azure and GCP services, SQL and NoSQL databases, HTTP, and change-data-capture from PostgreSQL, MySQL, MongoDB, SQL Server and Oracle
  • Bloblang makes the common job, mapping one document shape into another, short and testable
  • Custom inputs, processors and outputs can be written in Go as plugins against the public API

Watch out for

  • Licence split: CDC inputs, the Iceberg, Snowflake, BigQuery Write API and Splunk connectors, and features such as secrets management need a paid Enterprise key; the list was two connectors at the 2024 acquisition and has grown since
  • Not a stateful stream processor: no checkpointed state, no event-time watermarks; the system_window buffer follows the system clock, can’t backfill, and drops messages that miss their window
  • At-least-once, not exactly-once: after a crash or retry, outputs can see duplicates, and deduplication (for example with a cache) is your job
  • Rename churn: the Benthos name, repo and community moved to Redpanda, and components such as kafka_franz are deprecated in favour of the unified redpanda input and output

Alternatives

What all stream processing have in common →
Apache FlinkOpen sourceThe most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

Read orders from Kafka, reshape them with Bloblang, write gzipped batches to S3 · yaml

# rpk connect run orders.yaml
input:
  redpanda:
    seed_brokers: [ "kafka-1:9092", "kafka-2:9092" ]
    topics: [ orders ]
    consumer_group: orders-to-s3

pipeline:
  processors:
    - mapping: |
        root = if this.status == "test" { deleted() } else { this }
    - mapping: |
        root.order_id = this.id
        root.customer_id = this.customer.id
        root.total_cents = (this.total * 100).round()
        root.received_at = now()

output:
  aws_s3:
    bucket: orders-archive
    path: orders/${! counter() }-${! timestamp_unix_nano() }.jsonl.gz
    batching:
      count: 1000
      period: 30s
      processors:
        - archive:
            format: lines
        - compress:
            algorithm: gzip

Which licence covers what#

Checked against the repository and docs on 2026-10-05.

LicenceFree to use?Examples
Core engine (redpanda-data/benthos)MITYesPipeline runtime, config, Bloblang, the Go plugin API
Most connectors (redpanda-data/connect)Apache 2.0Yesredpanda (Kafka) input and output, aws_s3, SQS, NATS, MQTT, AMQP, HTTP, sql_insert, Redis, Elasticsearch
Enterprise componentsRedpanda Community LicenseOnly with an Enterprise key (30-day trial available)postgres_cdc, mysql_cdc, mongodb_cdc, microsoft_sql_server_cdc, oracledb_cdc, gcp_spanner_cdc, aws_dynamodb_cdc, iceberg, snowflake_streaming, gcp_bigquery_write_api, splunk_hec, Salesforce, secrets management, allow and deny lists
Bento (warpstreamlabs/bento)MITYesCommunity fork of Benthos from before the May 2024 licence change, maintained by WarpStream
  • The community build (redpanda-connect-community) leaves the enterprise components out; the standard binary includes them but blocks them until a licence key is set.

See it in a real stack

No architecture uses Redpanda Connect yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2015–2024

Moving data between systems is mostly plumbing: read from a queue, reshape each message, write it somewhere else, and lose nothing if the process crashes. Ashley Jeffs started Benthos in 2015 as a Go program for that job, with the tagline “fancy stream processing made operationally mundane”. Instead of storing messages on disk, it acknowledged each message to the source only after the output confirmed it, which gave at-least-once delivery with no state of its own. For years it was, in his words, “a mostly one-person show”, paid for indirectly through jobs, support contracts and sponsorships. Version 1.0 came out in January 2019, and Bloblang, its mapping language, in 2020.

On May 30, 2024, Redpanda acquired Benthos and renamed it Redpanda Connect. The core engine moved to a separate MIT-licensed repository, most connectors moved from MIT to Apache 2.0, and a few, starting with the Snowflake and Splunk outputs, went under the source-available Redpanda Community License. The next day WarpStream forked the pre-change code as Bento, objecting that integrations written by outside contributors had become proprietary.

Why it’s built this way

No disk state: at-least-once through acknowledgements
Holding back the acknowledgement to the source until the output confirms a message gives at-least-once delivery without persisting messages in transit, so an instance can be killed, moved or copied freely.
A whole stream in one config file
From 1.0 a Benthos stream was four layers in one file: inputs, an optional buffer, processors and outputs. Declaring the pipeline instead of coding it keeps the boring work boring.
Its own mapping language
JMESPath made small edits to a large document verbose and risky, and the AWK processor was a full language that was slower and easy to break. Jeffs designed Bloblang because mapping is “one of the most common boring tasks” in stream processing.
Also a Go framework
The same engine is a library, so teams can compile their own inputs, processors and outputs into a custom binary instead of waiting for a built-in connector.

Timeline

  1. 2015

    Ashley Jeffs makes the first commit to Benthos in September

  2. 2019

    Benthos 1.0 is released in January

  3. 2020

    Bloblang mapping language arrives in Benthos 3.12 and 3.13

  4. 2022

    Benthos 4.0 is released in April

  5. 2024

    Redpanda acquires Benthos and renames it Redpanda Connect; licences split

  6. 2024

    WarpStream forks the MIT code as Bento

Sources24 · facts checked 5 October 2026

Improve this page on GitHub