Cloud Parts

Apache Flink

The most widely used open-source engine for stateful stream processing: keyed state, event time and exactly-once recovery from checkpoints.

What it is
Stream processing
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
6 October 2026
Website Source code

What it is

A Flink job is a graph of operators (sources, maps, joins, windows, sinks) written in SQL, the Table API, the Java DataStream API or Python. A client hands the graph to the JobManager, which splits each operator into parallel subtasks and schedules them into the task slots of the TaskManager processes. Records are routed by key, so every event for one key reaches the same subtask, and that subtask keeps the key’s state: on the Java heap by default, in RocksDB on local disk for state bigger than memory, or, since Flink 2.0, experimentally in ForSt, which keeps state in object storage such as S3 with a local cache. Windows close by event time: a watermark says how far the event clock has got. To survive failures, the JobManager regularly injects checkpoint barriers at the sources; as they flow through the graph, each operator snapshots its state, and the source offsets are stored alongside it in durable storage. After a crash the job restarts from the last checkpoint and rereads the input from those offsets, and transactional sinks such as Kafka make the output exactly once. A savepoint is the same snapshot taken by hand, used to upgrade code or change parallelism.

Flink is an Apache project under Apache 2.0, and runs as a standalone cluster, on YARN or on Kubernetes, where the Flink Kubernetes Operator manages jobs as custom resources. Flink 2.0 (March 2025) was the first major release in nine years: it added disaggregated state and removed long-deprecated APIs such as DataSet and the Scala APIs; 2.3 followed in June 2026. Running it yourself is free but takes work, so three companies sell it as a service: AWS (Amazon Managed Service for Apache Flink, called Kinesis Data Analytics until 2023), Confluent (Confluent Cloud for Apache Flink, SQL over its Kafka topics) and Ververica, the company Flink’s creators founded, now with its own Flink-compatible engine.

Why teams pick it

  • Large, exactly-once state: keyed state can outgrow memory (RocksDB, or ForSt on object storage), and checkpoints store it together with the source offsets
  • Real event-time processing: watermarks, tumbling, sliding and session windows, timers and a CEP library for patterns across events
  • Savepoints let you stop a job, change its code or parallelism, and resume from exactly where it was
  • One engine for streams and bounded batch input, with SQL, Java and Python APIs on top
  • The widest ecosystem in stream processing: Kafka, Kinesis, Iceberg, Paimon, database change capture through Flink CDC, and three managed services

Watch out for

  • Running it is a job in itself: JobManager high availability, checkpoint storage, state backend and memory tuning, and upgrades through savepoints
  • Flink 2.x broke compatibility: state from 1.x isn’t guaranteed to restore, and the DataSet API, Scala APIs and old SourceFunction/SinkFunction connectors are gone
  • Connectors ship separately and lag the core: in October 2026 the Kafka connector supports Flink up to 2.2 and the AWS (Kinesis) connectors only 2.0, while Flink itself is at 2.3
  • Disaggregated state (ForSt) is still experimental, and Amazon’s managed service doesn’t offer it, or materialized tables
  • JVM first: Python user-defined functions run in separate Python processes by default, paying for serialization between the two

Alternatives

What all stream processing have in common →
ArroyoOpen sourceA stream processing engine in Rust: you write the pipeline as SQL, and it keeps windows and joins running with checkpointed state.BytewaxOpen sourceA Python stream processing framework on a Rust engine built on Timely Dataflow; its company wound down in 2025 and the last release is from November 2024.Cloudflare Basin PipelinesRenamedCommercialCloudflare’s managed streaming ingestion service, called Cloudflare Pipelines until October 2026, that takes events over HTTP or a Workers binding, transforms them with SQL and writes them to R2 as Apache Iceberg tables, Parquet or JSON.DenormalizedArchivedOpen sourceAn embeddable stream processing library in Rust, with Python bindings, built on Apache DataFusion. Abandoned since December 2024 at version 0.0.13.FelderaOpen sourceAn incremental view maintenance engine in Rust: you write SQL tables and views, and it keeps every view up to date by processing only the changes, based on the award-winning DBSP theory.Kafka StreamsOpen sourceA Java library, part of Apache Kafka, that turns your own application into a stateful stream processor with no cluster of its own.ksqlDBCommercialConfluent’s SQL server for Kafka: queries that run forever, turning topics into new topics and into tables you can look up.PathwayCommercialA Python framework for streaming ETL and live RAG pipelines: you write table transformations in Python, and a Rust engine built on Differential Dataflow keeps the results up to date.Quix StreamsOpen sourceA pure-Python library for stateful stream processing on Kafka: a pandas-like Streaming DataFrame, RocksDB state backed up to Kafka, and no cluster of its own.Redpanda ConnectRenamedOpen sourceFormerly Benthos: one Go binary that runs a YAML pipeline of inputs, processors and outputs, with at-least-once delivery and no disk state of its own.RisingWaveOpen sourceA streaming database in Rust that speaks the PostgreSQL protocol: you write materialized views in SQL, and it keeps them up to date as events arrive, with all state in object storage.StreamFusionOpen sourceAn open-source accelerator for Apache Flink SQL: Flink still plans and coordinates the job, while the operators it can reproduce exactly run natively in Rust on Arrow and DataFusion.StreamlingCommercialGoldsky’s young Rust pipeline runtime: YAML pipelines of stateless SQL, TypeScript and plugin steps on Arrow, from Kafka or files into Postgres, ClickHouse or webhooks.Timely DataflowOpen sourceA Rust library for data-parallel dataflow with logical timestamps, loops and exact progress tracking; the engine underneath Materialize.

How it fits with other technologies

What uses Apache Flink

Run a stateful job, stop it with a savepoint, and resume it with more parallelism · bash

./bin/start-cluster.sh            # a JobManager and one TaskManager on localhost

# a long-running example job with keyed state and a built-in data generator
./bin/flink run --detached ./examples/streaming/StateMachineExample.jar
./bin/flink list                  # note the JobID
export JOB_ID=cca7bc1061d61cf15238e92312c2fc20

# stop gracefully: sources emit a final barrier, state is written to a savepoint
./bin/flink stop --savepointPath /tmp/flink-savepoints $JOB_ID
# Savepoint completed. Path: file:/tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab

# add a second TaskManager (one slot each by default), then resume from that state
# with 2 subtasks per operator; the same JAR or a new version of it
./bin/taskmanager.sh start
./bin/flink run --detached --parallelism 2 \
  --fromSavepoint /tmp/flink-savepoints/savepoint-cca7bc-bb1e257f0dab \
  ./examples/streaming/StateMachineExample.jar

See it in a real stack

No architecture uses Apache Flink yet. Drop it into the playground and wire it up yourself.Open playground →

Worth exploring

Article· September 2025Understanding Apache FlussJack VanlightlyA long, detailed walk through Fluss internals, reverse-engineered from the code: log tables, primary-key tables that emit changelogs, and tiering to lakehouse tables. The author works at Confluent, which runs Flink but not Fluss, and says so up front.
Article· October 2024Stream Processing With Apache Flink: book notesChris Shumaker, Maker ConsultingOne engineer’s chapter-by-chapter notes on the O’Reilly book by Fabian Hueske and Vasiliki Kalavri, read to understand what owning a Flink job in production involves: state, time, scale and performance. A quick way to decide whether the book is worth reading in full.
All resources →
Where it came from2009–2026

In 2009 researchers at the Technical University of Berlin started Stratosphere, a research project to build the next generation of big data analytics platform. Its motto was “write like a programming language, execute like a database”: it mixed ideas from MapReduce and parallel databases, and managed its own memory as binary data instead of piling Java objects on the heap.

Stratosphere’s engine entered the Apache Incubator as Flink in April 2014 and graduated in December. The same year its core developers, among them Kostas Tzoumas and Stephan Ewen, founded data Artisans in Berlin to build a business around it, and in 2015 they turned Flink towards stream processing. Version 0.10 (November 2015) brought event time, out-of-order streams and exactly-once recovery; 1.0 (March 2016) added RocksDB state and savepoints. Alibaba, which ran its own fork called Blink, bought data Artisans for €90 million in January 2019 and contributed Blink back to the project; the company is now called Ververica.

Flink became the default open-source engine for stateful streaming, and cloud providers built services on it: AWS added Flink to Kinesis Data Analytics in 2018 and renamed the service after Flink in 2023. Flink 2.0, in March 2025, was the first major release in nine years.

Why it’s built this way

Streams first, batch as a stream that ends
If an engine can process unbounded streams continuously, bounded data sets are just streams that stop. Building on that idea let one runtime and, over time, one API serve both real-time and historical processing.
Asynchronous barrier snapshots for fault tolerance
Earlier approaches either paused the computation to take a global snapshot or saved every in-flight record. Barriers that flow with the data let each operator snapshot only its own state without stopping, which Flink’s developers described in a 2015 paper.
Managing its own memory as binary data
From its database roots, Flink serialized records into pre-allocated memory segments and ran sort and join algorithms directly on the bytes. That avoided out-of-memory crashes and heavy garbage collection on large JVM heaps, and let operators spill to disk predictably.
Event time and watermarks
Events often arrive late and out of order. Grouping by when things happened, with watermarks marking progress, gives the same results on live data and on a replay, which 0.10 made a core feature.

Timeline

  1. 2009

    Stratosphere research project starts at TU Berlin

  2. 2014

    Enters the Apache Incubator as Flink in April, top-level in December; data Artisans founded

  3. 2016

    Flink 1.0 adds RocksDB state and savepoints, and promises API compatibility

  4. 2019

    Alibaba buys data Artisans (now Ververica) and contributes its Blink fork

  5. 2023

    Kinesis Data Analytics becomes Amazon Managed Service for Apache Flink

  6. 2025

    Flink 2.0: disaggregated state; DataSet and Scala APIs removed

  7. 2026

    Flink 2.3 adds changelog SQL operators and a native S3 filesystem

Sources29 · facts checked 6 October 2026

Improve this page on GitHub