Analytic database on the FDAP stack
Arrow Flight, DataFusion, Arrow and Parquet as the reusable core of a database, the way InfluxDB 3 is built.
Click any box to open its page. Open it in the playground to rewire it, change the traffic or take parts down.
Writing an analytic database used to mean writing everything: a file format, an in-memory layout, a query planner and engine, and a network protocol with clients for every language. The FDAP stack, named by InfluxData in 2023, takes those four layers from Apache projects instead: Arrow Flight to move data over the network, DataFusion to plan and run queries, Arrow as the in-memory columnar format, and Parquet for files on object storage. The team then writes only what is specific to its domain. This page follows InfluxDB 3 Core, the best-known system built this way: time-series writes land in an Arrow buffer, are saved as Parquet in a bucket, and are queried through DataFusion and returned over Flight.
How it flows
- 1
Writes are validated and logged
Agents such as Telegraf send line protocol over HTTP. InfluxDB 3 validates each write, adds it to an in-memory buffer and, by default every second, flushes that buffer as a write-ahead log file to object storage. Once the WAL file is stored, the write is durable.
- 2
Recent data stays in memory as Arrow
Logged data moves into a queryable buffer of Arrow record batches, the same columnar layout the query engine works on, so new points can be queried immediately without being converted.
- 3
Older data is persisted as Parquet
Every ten minutes by default the oldest data in the buffer is written as Parquet files to the bucket. Parquet is the long-term format, and because it is an open standard, Spark, DuckDB, pandas or Snowflake can read the same files without an export pipeline.
- 4
DataFusion plans one query over both
A SQL or InfluxQL query becomes one DataFusion plan that scans the Arrow buffer and the Parquet files together, skipping files and row groups whose statistics can’t match. InfluxData added its InfluxQL front end, deduplication and gap-filling operators through DataFusion’s extension points.
- 5
Results stream back over Flight
The client asks for a result with Flight SQL, a database protocol on top of Arrow Flight, and receives a stream of Arrow record batches over gRPC. Nothing is turned into rows or JSON on the way, so a Python client gets an Arrow table it can hand straight to pandas. An HTTP API with JSON, CSV or Parquet output is there for clients without Flight.
Why this shape works
- The hard parts are shared: InfluxData’s engineers estimated a DataFusion-class query engine at tens of person-years, and building on it brought features such as SQL joins almost for free
- Open formats end to end: Parquet files and Arrow results work with pandas, Spark, DuckDB and BI tools without custom exporters
- Flight and Flight SQL give clients in many languages, and let existing JDBC, ODBC and Python DB-API tools connect, without a team writing drivers
- All four parts are Apache projects with many contributors, so their licences and direction can’t be changed overnight by one company
Where it strains
- You still build the database: the WAL, catalog, compaction, retention and clustering are yours, and InfluxDB 3 Core, for example, ships without a compactor
- Much of the stack is Rust; DataFusion’s extension APIs are Rust traits, and its major versions, every few months, break those APIs
- Parquet is a general format. InfluxData later gave InfluxDB 3 Enterprise its own time-series file format by default for new clusters and kept Parquet for export, a sign that a specialised format can still win for one workload
- Flight is gRPC, not plain HTTP, so InfluxDB 3 also keeps an HTTP query API returning JSON, CSV or Parquet for clients that can’t use it
Common swaps
Need a finished database, not parts? GreptimeDB and InfluxDB 3 Core are both built on the same pieces.
Keep Parquet on Google Cloud Storage, Azure Blob Storage or local disk instead of S3; the object-store layer is the same.
Want shared tables with snapshots instead of your own file layout? delta-rs and Lance both plug into DataFusion.
Embedding analytics in one process rather than building a server? DuckDB plays DataFusion’s role and reads and returns Arrow too.
Built from
InfluxDBOpen sourceTime-series databasesThe best-known time-series database, fed by line protocol and Telegraf; now in its third generation, a Rust rewrite that keeps data as Parquet files on object storage.
Apache ParquetOpen sourceFile formatsThe standard columnar file format for analytics: each column stored and compressed on its own, with statistics that let engines skip data.
Apache ArrowOpen sourceFile formatsA standard way to lay out columnar data in memory, so libraries, engines and languages can share tables without copying or converting them.
Amazon S3AWSObject storageAWS’s object storage, and the de facto API most object stores copy.
Apache DataFusionOpen sourceQuery enginesA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.Architectures with parts in common
Sources
- Andrew Lamb, InfluxData: Flight, DataFusion, Arrow and Parquet, using the FDAP architecture to build InfluxDB 3.0
- Paul Dix, InfluxData: Announcing InfluxDB IOx (Rust, Arrow, Parquet, DataFusion, object storage)
- InfluxData: FDAP stack explained (term coined in 2023; roles of the four parts)
- Data Engineering Podcast, episode 414: Paul Dix on the FDAP stack (February 2024)
- InfluxDB 3 Core docs: Data durability (WAL every second, queryable buffer, Parquet every ten minutes)
- InfluxDB 3 Core docs: Query data (Flight and HTTP APIs, Arrow tables in Python, output formats)
- InfluxDB 3 Enterprise docs: Storage engine (.pt files by default for new clusters, Parquet downgrade)
- Apache Arrow docs: Arrow Flight RPC (GetFlightInfo, DoGet, gRPC)