Apache DataFusion

A query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.

What it is: Query engines
Type: Open source
Made by: Apache Software Foundation
License: Apache 2.0
Facts checked: 5 October 2026

What it is

DataFusion is a query engine you compile into your own program, not a server you connect to. You hand it SQL or build a query with its DataFrame API. It parses and plans the query, rewrites the plan with its optimiser, then runs it on a streaming, multi-threaded, vectorised engine that passes Apache Arrow record batches between operators. It keeps no storage of its own: it reads Parquet, CSV, JSON and Avro files from local disk or object storage such as S3, in-memory Arrow data, or any source you plug in, and writes results back with COPY ... TO or INSERT INTO.

Almost every layer can be replaced: table sources (the TableProvider trait), scalar, aggregate and window functions, optimiser rules, and logical and physical plan nodes. That makes it a kit for building data systems, and dozens do: InfluxDB 3, the Arroyo stream processor, delta-rs, Lance, GreptimeDB, Spice.ai, Cloudflare’s R2 query engine and the dbt Fusion engine among them. Subprojects add a command-line client (datafusion-cli), Python bindings (pip install datafusion), Ballista for running queries across several machines, and Comet, a plugin that runs Apache Spark jobs on DataFusion. A new major version ships every two to three months; the current one is 55.1.0.

Why teams pick it

A full SQL planner, optimiser and vectorised engine you can embed instead of writing your own
Extension points at every layer: data sources, functions, optimiser rules and execution operators
Works on Arrow, so results pass to other Arrow tools in Python, Rust and elsewhere without conversion
Apache 2.0 licensed and governed by the ASF, with many companies building on and contributing to it
Reads Parquet efficiently from object storage, skipping row groups and pages that can’t match

Watch out for

It’s a library, not a database: no server, user management or persistent catalog unless you build them or use a product built on it
Customising it means writing Rust; the extension APIs are Rust traits
Each major release can break the Rust API, and majors arrive every few months, so embedders must budget for upgrades
One process by default; running across machines needs Ballista or a distributed layer of your own

The Apache DataFusion family

1 technologies built from or on Apache DataFusion

Built on it

Apache DataFusion CometOpen sourceA Spark plugin that keeps Spark’s optimizer and scheduler but runs the physical plan on Apache DataFusion in Rust, over Arrow batches, with no changes to your Spark code, and falls back to Spark for anything it can’t run the same way.

Alternatives

What all query engines have in common →

Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.

Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.

Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.

Apache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.

Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.

Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.

Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.

chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.

Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.

DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.

GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.

GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.

HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.

IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.

PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.

Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.

RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.

Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.

TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.

VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

What Apache DataFusion uses

What uses Apache DataFusion

The same query in SQL and with the DataFrame API · python

from datafusion import SessionContext, col, lit
from datafusion import functions as f

ctx = SessionContext()
ctx.register_parquet("trips", "data/trips/")

ctx.sql("""
    SELECT pickup_zone, count(*) AS trips, avg(fare) AS avg_fare
    FROM trips WHERE fare > 0
    GROUP BY pickup_zone ORDER BY trips DESC LIMIT 10
""").show()

# a lazy plan; nothing runs until results are requested
top = (ctx.table("trips")
       .filter(col("fare") > lit(0))
       .aggregate([col("pickup_zone")], [f.count(col("fare")).alias("trips")])
       .sort(col("trips").sort(ascending=False))
       .limit(10))

table = top.to_arrow_table()  # results as a PyArrow table

The DataFusion family#

DataFusion

Rust library

The core: SQL and DataFrame APIs, planner, optimiser and execution engine, added to a Rust project as a dependency.

datafusion-cli

command line

An interactive SQL shell. Queries files by path, including s3:// and gs:// locations, or as external tables.

DataFusion Python

pip install datafusion

Python bindings with SQL and a lazy DataFrame API. Reads pandas, Polars and Arrow data, and returns results in any of them.

Ballista

distributed

A scheduler and executors that spread DataFusion queries across many machines. Andy Grove donated it to Apache Arrow in 2021.

Comet

Spark plugin

Translates Spark physical plans into DataFusion plans and runs them natively, with no code changes. Started at Apple, donated in 2024, 1.0 in August 2026.

DataFusion or DuckDB?#

Both are fast, embeddable, columnar SQL engines that run on one machine. They are aimed at different people.

	DataFusion	DuckDB
What you get	An engine to build on	A finished database to use
Storage	None of its own; reads files and plugged-in sources	Its own single-file database, plus files in place
Customise by	Replacing parts in Rust: sources, functions, optimiser rules, operators	Loading extensions
Runs	Inside your Rust or Python program; any server is yours to build	Inside your program, or as a server through its Quack protocol
Licence	Apache 2.0	MIT
Typical user	Teams building a database, stream processor or data service	Analysts and engineers querying data from Python, R or the CLI

See it in a real stack

No architecture uses Apache DataFusion yet. Drop it into the playground and wire it up yourself.Open playground →

Benchmarks

Benchmark · ClickHouse Inc.ClickBenchHow fast analytical databases run 43 SQL queries over one 100-million-row table of web analytics events, on the same cloud machines.Includes Apache DataFusion · 7 caveats to read first

How to read a benchmark →

Where it came from2017–2026

Apache Spark ran on the JVM, and much of its engineering went into working around that: Java serialization between nodes, garbage collection and out-of-memory errors that took constant tuning. Andy Grove noted that Spark’s Project Tungsten was, in effect, “making a JVM product make less use of the JVM”. In 2017 he started DataFusion to test a hypothesis: a Spark-like engine written in Rust would be fast even when naive, and more predictable and reliable.

After rebuilding it on Arrow’s Rust library, Grove donated DataFusion to the Apache Arrow project in February 2019. It could then run simple SQL over CSV files. In 2021 he donated Ballista, a distributed engine built on it. Companies that needed a query engine began building on DataFusion instead of writing one: InfluxData chose it for InfluxDB 3.0 rather than spend what its engineers estimated at tens of person-years building another engine, and Apple donated Comet, its Spark accelerator, in 2024.

In April 2024 DataFusion left Arrow to become a top-level Apache project, as more contributors worked on it alone rather than alongside the Arrow Rust library.

Why it’s built this way

Rust instead of the JVM: Grove saw Spark’s performance and reliability problems as coming from Java serialization and garbage collection. Rust gives native speed and predictable memory use without a garbage collector.
Arrow as the in-memory format: The 2019 rewrite made DataFusion Arrow-native, so data passes between DataFusion, Arrow libraries and other Arrow-based engines without copying or conversion.
A library with extension points everywhere: A complete engine takes tens of person-years. Making sources, functions, optimiser rules and operators replaceable lets projects such as InfluxDB reuse that work and add only what is specific to them.

Timeline

2017
Andy Grove starts DataFusion, a query engine in Rust
2019
Rebuilt on Arrow and donated to the Apache Arrow project in February
2021
Grove donates Ballista, a distributed scheduler built on DataFusion
2024
Apple donates Comet in March; top-level Apache project in April
2026
Comet 1.0 in August; DataFusion 55.0.0 the same month

Sources24 · facts checked 5 October 2026

Improve this page on GitHub