Apache Arrow AceroExperimental
The streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.
- What it is
- Query engines
- Type
- Open source
- Made by
- Apache Software Foundation
- License
- Apache 2.0
- Status
- Experimental. Arrow’s docs still mark it experimental, with no stable API guarantee.
- Facts checked
- 11 October 2026
What it is
Acero is part of Arrow C++ and builds on Arrow’s compute functions. A compute function works on arrays that are already in memory; Acero applies such functions to streams of batches. A plan is a graph of Declaration objects, one per node: sources (an in-memory table, a record batch reader, or a dataset scan over Parquet, CSV or other files), compute nodes (filter, project, aggregate, pivot_longer), arrangement nodes (hashjoin, asofjoin, union, order_by, fetch) and sinks (collect into a table, hand back a reader with backpressure, or write files). Data moves in small chunks: a 2-million-row input can come out as 64 batches of 32Ki rows. Plans can also arrive as Substrait, a cross-engine format for query plans, which the docs call the preferred way to build them.
Acero is deliberately narrow. It has no SQL parser, query planner or optimizer and runs a plan exactly as written; its authors want optimization done elsewhere and fed in through Substrait. It isn’t distributed, though a distributed engine could use it as a worker, and its writes have no transactions. Most people use it without knowing: Arrow’s dataset scanning and writing run on it, R’s arrow package uses it to run dplyr code, and in Python Table.join, Table.group_by and expression filters in pyarrow build Acero plans. Since Arrow 12 it is a separate library, libarrow_acero, exposed in Python as pyarrow.acero. It ships with every Arrow release (26.0.0, 9 October 2026), but its docs still say it is experimental and has no stable API.
Why teams pick it
- Already installed wherever Arrow C++ or
pyarrowis, and works on Arrow data directly, so there is nothing to convert or copy - Streams data in batches with backpressure, so a plan can feed a reader or write partitioned files without collecting the whole result in memory
- Accepts Substrait plans, so a front end that emits Substrait (for example Ibis through
ibis-substrait) can target Acero and later switch engines - Has an as-of join node, which joins several inputs to the first one on a shared ordered column, usually time
Watch out for
- No SQL and no optimizer: you choose every node, join order and filter position yourself, or use a front end that does
- Runs in one process; distributing work across machines is up to whatever embeds it
- Marked experimental with no stable API guarantee, even though it is used inside
pyarrowand Arrow’s dataset layer - Writes have no transaction support, and results collected into a table come back in Acero’s own chunk sizes (a request to control them, GH-15155, is still open)
Alternatives
What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.
Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.
Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.
GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.
HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.
IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.
Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.
VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.How it fits with other technologies
Spell out a filter, hash join, aggregate and sort as an Acero plan in pyarrow · python
# pip install pyarrow==26.0.0
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.acero import (Declaration, TableSourceNodeOptions, FilterNodeOptions,
HashJoinNodeOptions, AggregateNodeOptions, OrderByNodeOptions)
orders = pa.table({"order_id": [1, 2, 3, 4, 5], "customer_id": [10, 20, 10, 30, 20],
"amount": [25.0, 40.0, 15.0, 90.0, 5.0]})
customers = pa.table({"customer_id": [10, 20, 30], "country": ["DE", "US", "DE"]})
# Acero has no SQL parser or optimizer: you spell out every node of the plan
plan = Declaration.from_sequence([
Declaration("table_source", TableSourceNodeOptions(orders)),
Declaration("filter", FilterNodeOptions(pc.field("amount") > 10)),
])
plan = Declaration("hashjoin", HashJoinNodeOptions("inner", ["customer_id"], ["customer_id"]),
inputs=[plan, Declaration("table_source", TableSourceNodeOptions(customers))])
plan = Declaration.from_sequence([
plan,
Declaration("aggregate", AggregateNodeOptions(
[("amount", "hash_sum", None, "revenue"), ("amount", "hash_count", None, "orders")],
keys=["country"])),
Declaration("order_by", OrderByNodeOptions([("revenue", "descending")])),
])
print(plan.to_table().to_pylist())
# [{'country': 'DE', 'revenue': 130.0, 'orders': 3}, {'country': 'US', 'revenue': 40.0, 'orders': 1}]
# pyarrow's Table.join and Table.group_by build the same kind of plan for you
print(orders.join(customers, "customer_id").group_by("country")
.aggregate([("amount", "sum")]).sort_by("country").to_pylist())
# [{'country': 'DE', 'amount_sum': 130.0}, {'country': 'US', 'amount_sum': 45.0}]Acero’s building blocks#
The node types listed in the Acero user’s guide, checked 2026-10-11.
| Kind | Nodes | What they do |
|---|---|---|
| Sources | table_source, record_batch_source, record_batch_reader_source, exec_batch_source, array_vector_source, source, scan | Feed batches in from memory, a reader, a custom async generator, or files through Arrow Datasets |
| Compute | filter, project, aggregate, pivot_longer | Drop rows, compute columns, summarize (grouped or not), turn columns into rows |
| Arrangement | hashjoin, asofjoin, union, order_by, fetch | Join, merge, reorder or slice streams |
| Sinks | sink, table_sink, consuming_sink, write | Usually chosen for you by DeclarationToTable, DeclarationToReader or DeclarationToStatus |
See it in a real stack
No architecture uses Apache Arrow Acero yet. Drop it into the playground and wire it up yourself.Open playground →Where it came from2021–2026
Arrow is a shared in-memory columnar format, and its C++ library grew compute functions and a datasets layer for reading collections of files. The compute functions work on data that is already in memory; scanning many files, joining and grouping needed a way to stream batches through a series of operations. Arrow 5.0.0 in July 2021 added an ExecNode hierarchy, wrapped grouped aggregation in a node and rebuilt dataset scans as graphs of these nodes. Arrow 6.0.0 in October 2021 shipped what its release post called a basic in-memory query engine, with filter, project, sort, equality joins and aggregations, and R users got grouped aggregation and joins in dplyr on top of it.
In May 2022 the developers reached consensus on a name, Acero, and from Arrow 9.0.0 (August 2022) the docs call it “Acero: A C++ streaming execution engine”. Arrow 12.0.0 in May 2023 moved Acero out of the core libarrow into its own shared library, so the core build could shrink, added the order_by, fetch and pivot_longer nodes, and gave Python a pyarrow.acero module. Its documentation sets out what it is not: not a database, not an optimizer, not distributed, not a tool for data scientists to use directly.
Why it’s built this way
- No parser, planner or optimizer
- The docs call creating the best plan very hard and argue the optimizer should live outside Acero, composed through standards such as Substrait so any backend can use it.
- Streams of batches, not tables
- Working on one batch at a time removes the need for multi-batch containers and avoids awkward cases such as table columns chunked at different sizes.
- Substrait as the preferred input
- Substrait producers offer friendlier APIs than hand-built node graphs, and a plan in a standard format can move to another Substrait-consuming engine.
- A separate library from Arrow 12
- Splitting
libarrow_aceroout oflibarrowlets projects that only need the format and compute functions ship smaller builds.
Timeline
- 2021
Arrow 5.0.0 (July) adds the ExecNode hierarchy and runs dataset scans as node graphs
- 2021
Arrow 6.0.0 (October) ships a basic query engine, used by R’s dplyr bindings
- 2022
Developers settle on the name Acero in May; Arrow 9.0.0 docs use it in August
- 2023
Arrow 12.0.0 (May) splits out libarrow_acero and adds the pyarrow.acero module
- 2026
Arrow 26.0.0 (9 October) still ships Acero, documented as experimental
- Apache Arrow: 5.0.0 release notes (29 July 2021; ExecNode hierarchy, dataset scans as ExecNode graphs)
- Apache Arrow: 6.0.0 release notes (26 October 2021)
- Apache Arrow blog: 6.0.0 release (basic in-memory query engine, dplyr joins and aggregation)
- Apache Jira: ARROW-16632, “Given consensus on Acero as the name” (23 May 2022)
- Apache Arrow 8.0 docs: Streaming execution engine (before the name)
- Apache Arrow 9.0 docs: Acero, a C++ streaming execution engine
- Apache Arrow: 9.0.0 release notes (3 August 2022)
- Apache Arrow blog: 12.0.0 release (Acero in its own library, new nodes, pyarrow.acero; May 2023)
- Apache Arrow docs: Acero overview (what Acero is not, design)
Sources8 · facts checked 11 October 2026
- Apache Arrow docs: Acero overview (not a database, optimizer or distributed engine; ExecNode, ExecBatch, Declaration)
- Apache Arrow docs: Acero user’s guide (Substrait, DeclarationToXyz, node list, 32Ki-row batches)
- Apache Arrow docs: Acero, a C++ streaming execution engine (experimental, no stable API)
- Apache Arrow docs: Using Acero with Substrait
- Apache Arrow docs: pyarrow.acero API reference
- GitHub: apache/arrow python/pyarrow/acero.py (Table.join, group_by and filter built on Acero declarations)
- Apache Arrow blog: 12.0.0 release (libarrow_acero, pyarrow.acero)
- Apache Arrow: 26.0.0 release notes (9 October 2026; Acero fixes)