Apache NiFi

A Java dataflow server you program by drawing a graph of processors in a web UI, with buffered queues, back pressure and a record of where every piece of data went.

What it is: ETL & ELT pipelines
Type: Open source
Made by: Apache Software Foundation
License: Apache 2.0
Facts checked: 6 October 2026

What it is

NiFi runs in a JVM and serves its web UI and REST API from the same process. You build a flow on a canvas: processors (hundreds ship with it, such as ConsumeKafka, CaptureChangeMySQL, PutS3Object and PutDatabaseRecord) joined by connections. Each piece of data moving through is a FlowFile: a map of string attributes plus a content stream of zero or more bytes. Every connection is a queue with limits, by default 10,000 FlowFiles or 1 GB, and when a queue hits its limit the upstream processor stops being scheduled. That is back pressure. A flow controller hands threads to processors. Three disk repositories hold the state: a write-ahead log of the FlowFiles currently in the flow, a content repository with the bytes, and a provenance repository that indexes every event (created, modified, routed, sent) so you can trace an object’s lineage, view its content at any step and replay it. Provenance is kept for 30 days or 10 GB by default.

Changes on the canvas take effect immediately, without a deploy step. In a cluster every node runs the same flow on a different share of the data, with a coordinator elected through ZooKeeper or, since NiFi 2, Kubernetes. MiNiFi is a small agent, in Java or as a native C++ binary, that runs a subset of processors next to sensors and servers and forwards data to NiFi. Flows are versioned through Flow Registry Clients. Since NiFi 2 these can store flows directly in GitHub or GitLab, and the separate NiFi Registry service is deprecated. NiFi 2.0 (November 2024) requires Java 21, added a Python processor API (still labelled beta) and removed many old components.

Why teams pick it

Visual, live editing: you can see queues fill up and change a running flow one component at a time
Built-in provenance records the lineage of every FlowFile, which helps with audits, debugging and replaying data
Bounded queues with back pressure and prioritisation let slow destinations push back instead of losing data
A large library of processors for files, SFTP, HTTP, Kafka, databases, cloud storage and queues, extensible in Java or Python
Runs anywhere from a single Docker container to a cluster, with MiNiFi agents at the edge

Watch out for

Flows live in a canvas, not code: reviewing and testing changes is harder than with SQL or Python pipelines, and large flows become hard to follow
Running a cluster is real work: JVM heap tuning, several disk repositories, ZooKeeper or Kubernetes coordination, TLS and user management
It routes and reshapes data in flight; modelling inside the warehouse (SQL models and their tests) is a job for tools like dbt
Upgrades break things: NiFi 1.x lost support in December 2024, 2.0 removed many processors (HBase, Kudu, Kafka 2.6 and others), and NiFi Registry is now deprecated with removal planned for 3.0
Python processors are still marked beta and not recommended for production

Alternatives

What all etl & elt pipelines have in common →

AirbyteCommercialAn ELT platform you can host yourself: hundreds of connectors, each a Docker image speaking one JSON protocol, scheduled by services on Kubernetes.

ArtieCommercialA managed change data capture service that copies every insert, update and delete from production databases into a warehouse, aiming for under a minute of delay.

BruinOpen sourceOne Go command-line tool that ingests data, runs SQL and Python transformations and checks the results, with each step a file in your repo.

DagsterOpen sourceA Python data orchestrator built around assets: you declare the tables, files and models that should exist, and Dagster works out what to run.

dbtOpen sourceSQL `select` statements in git, built into warehouse tables in dependency order, with tests and docs alongside.

EstuaryCommercialA managed data-movement platform that captures changes from databases and SaaS sources into collections kept as JSON files in your own bucket, then keeps warehouses and other systems in sync, in real time or on a schedule.

FivetranCommercialManaged connectors that copy data from SaaS apps, databases and files into your warehouse or lake, billed by the rows that change each month.

MeltanoOpen sourceA command-line ELT tool that runs Singer taps and targets from a `meltano.yml` project kept in git.

OLakeOpen sourceAn open-source tool in Go that copies databases, Kafka topics and S3 files into Apache Iceberg tables (or plain Parquet), with full loads, incremental syncs and CDC, and no Kafka, Debezium, Spark or Flink in between.

How it fits with other technologies

What Apache NiFi uses

What uses Apache NiFi

Apache ZooKeepercan elect the cluster coordinator forApache NiFiCoordination services

Start NiFi 2 in Docker and ask the REST API how much data is queued · bash

docker run --name nifi -p 8443:8443 -d \
  -e SINGLE_USER_CREDENTIALS_USERNAME=admin \
  -e SINGLE_USER_CREDENTIALS_PASSWORD=correct-horse-battery \
  apache/nifi:2.12.0

# the canvas is at https://localhost:8443/nifi (self-signed certificate)

# the UI uses the same REST API: get a token, then read the controller status
TOKEN=$(curl -sk -X POST https://localhost:8443/nifi-api/access/token \
  --data-urlencode username=admin \
  --data-urlencode password=correct-horse-battery)

curl -sk -H "Authorization: Bearer $TOKEN" \
  https://localhost:8443/nifi-api/flow/status
# {"controllerStatus":{"activeThreadCount":2,"flowFilesQueued":1520,"bytesQueued":...}}

The NiFi family#

What the project ships as of NiFi 2.12.0 (September 2026).

NiFi

the server

The JVM process with the canvas, REST API, flow controller and the FlowFile, content and provenance repositories. Runs alone or as a cluster.

NiFi Stateless

separate binary

A separate download for running a flow in stateless mode. Since 2.0, process groups inside a normal NiFi can also use a stateless execution mode.

MiNiFi

edge agent

A small agent in Java, or a native C++ build for Linux and Windows with a subset of processors, that collects data at the source and forwards it to NiFi with provenance intact.

Git-based Flow Registry Clients

recommended

Store versioned flows directly in GitHub or GitLab repositories, with branches since 2.10.

NiFi Registry

deprecated

The separate flow-versioning service. Deprecated by community vote in February 2026 and planned for removal in NiFi 3.0.

NiFi Toolkit

CLI

Command-line tools, including the NiFi CLI for managing nodes, flows and registry clients.

See it in a real stack

No architecture uses Apache NiFi yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2006–2026

Intelligence agencies move huge volumes of data between networks that use different formats and protocols, under strict rules about who may see what. In late 2006 a team at the US National Security Agency, led by Joe Witt, started building a tool for this, called Niagarafiles. It was based on flow-based programming: small black-box processors joined by bounded queues, managed live through a visual interface, with a record of where every piece of data came from.

In November 2014 the NSA released it through its Technology Transfer Program, its first in a series of open-source releases, and it entered the Apache Incubator as NiFi. It became a top-level Apache project in July 2015. Hortonworks agreed in August 2015 to buy Onyara, whose engineers had been the key contributors to the government project, and built Hortonworks DataFlow on NiFi. Cloudera took that business over when it merged with Hortonworks in 2019.

NiFi made dataflow something you could watch and change while it ran. A decade on, its foundations had aged: NiFi 1 was stuck on Jetty 9.4, Spring 5.3 and AngularJS 1.8, so NiFi 2.0 (2024) moved to Java 21, rebuilt the UI and dropped many old components.

Why it’s built this way

Flow-based programming with bounded queues: Processors that only talk through queues are naturally concurrent and asynchronous, and capping the queues gives back pressure, so a slow system pushes back instead of losing data.
Change the flow while it runs: Production environments often can’t be reproduced in a lab, so NiFi treats a flow like clay: each change applies immediately to one component without stopping the rest.
Record provenance for every FlowFile: Compliance and security rules demand knowing where data came from and where it went, so lineage and content history are part of the core, not an add-on.
Separate attributes from content: Processors can route on a FlowFile’s attributes without reading its content, while the bytes stay in the content repository, which uses copy-on-write.

Timeline

2006
Development of Niagarafiles begins at the NSA
2014
NSA releases it as open source; Apache Incubator accepts NiFi in November
2015
Top-level Apache project in July; Hortonworks agrees to buy Onyara in August
2016
NiFi 1.0: new UI and clustering without a master node
2019
Cloudera completes its merger with Hortonworks
2024
NiFi 2.0: Java 21, Python processors, many old components removed
2026
NiFi Registry deprecated in favour of Git-based registry clients

Sources16 · facts checked 6 October 2026

Improve this page on GitHub