Cloud Parts

Apache Arrow AceroExperimental

The streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.

What it is
Query engines
Type
Open source
License
Apache 2.0
Status
Experimental. Arrow’s docs still mark it experimental, with no stable API guarantee.
Facts checked
11 October 2026

What it is

Acero is part of Arrow C++ and builds on Arrow’s compute functions. A compute function works on arrays that are already in memory; Acero applies such functions to streams of batches. A plan is a graph of Declaration objects, one per node: sources (an in-memory table, a record batch reader, or a dataset scan over Parquet, CSV or other files), compute nodes (filter, project, aggregate, pivot_longer), arrangement nodes (hashjoin, asofjoin, union, order_by, fetch) and sinks (collect into a table, hand back a reader with backpressure, or write files). Data moves in small chunks: a 2-million-row input can come out as 64 batches of 32Ki rows. Plans can also arrive as Substrait, a cross-engine format for query plans, which the docs call the preferred way to build them.

Acero is deliberately narrow. It has no SQL parser, query planner or optimizer and runs a plan exactly as written; its authors want optimization done elsewhere and fed in through Substrait. It isn’t distributed, though a distributed engine could use it as a worker, and its writes have no transactions. Most people use it without knowing: Arrow’s dataset scanning and writing run on it, R’s arrow package uses it to run dplyr code, and in Python Table.join, Table.group_by and expression filters in pyarrow build Acero plans. Since Arrow 12 it is a separate library, libarrow_acero, exposed in Python as pyarrow.acero. It ships with every Arrow release (26.0.0, 9 October 2026), but its docs still say it is experimental and has no stable API.

Why teams pick it

  • Already installed wherever Arrow C++ or pyarrow is, and works on Arrow data directly, so there is nothing to convert or copy
  • Streams data in batches with backpressure, so a plan can feed a reader or write partitioned files without collecting the whole result in memory
  • Accepts Substrait plans, so a front end that emits Substrait (for example Ibis through ibis-substrait) can target Acero and later switch engines
  • Has an as-of join node, which joins several inputs to the first one on a shared ordered column, usually time

Watch out for

  • No SQL and no optimizer: you choose every node, join order and filter position yourself, or use a front end that does
  • Runs in one process; distributing work across machines is up to whatever embeds it
  • Marked experimental with no stable API guarantee, even though it is used inside pyarrow and Arrow’s dataset layer
  • Writes have no transaction support, and results collected into a table come back in Acero’s own chunk sizes (a request to control them, GH-15155, is still open)

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

Spell out a filter, hash join, aggregate and sort as an Acero plan in pyarrow · python

# pip install pyarrow==26.0.0
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.acero import (Declaration, TableSourceNodeOptions, FilterNodeOptions,
                           HashJoinNodeOptions, AggregateNodeOptions, OrderByNodeOptions)

orders = pa.table({"order_id": [1, 2, 3, 4, 5], "customer_id": [10, 20, 10, 30, 20],
                   "amount": [25.0, 40.0, 15.0, 90.0, 5.0]})
customers = pa.table({"customer_id": [10, 20, 30], "country": ["DE", "US", "DE"]})

# Acero has no SQL parser or optimizer: you spell out every node of the plan
plan = Declaration.from_sequence([
    Declaration("table_source", TableSourceNodeOptions(orders)),
    Declaration("filter", FilterNodeOptions(pc.field("amount") > 10)),
])
plan = Declaration("hashjoin", HashJoinNodeOptions("inner", ["customer_id"], ["customer_id"]),
                   inputs=[plan, Declaration("table_source", TableSourceNodeOptions(customers))])
plan = Declaration.from_sequence([
    plan,
    Declaration("aggregate", AggregateNodeOptions(
        [("amount", "hash_sum", None, "revenue"), ("amount", "hash_count", None, "orders")],
        keys=["country"])),
    Declaration("order_by", OrderByNodeOptions([("revenue", "descending")])),
])

print(plan.to_table().to_pylist())
# [{'country': 'DE', 'revenue': 130.0, 'orders': 3}, {'country': 'US', 'revenue': 40.0, 'orders': 1}]

# pyarrow's Table.join and Table.group_by build the same kind of plan for you
print(orders.join(customers, "customer_id").group_by("country")
      .aggregate([("amount", "sum")]).sort_by("country").to_pylist())
# [{'country': 'DE', 'amount_sum': 130.0}, {'country': 'US', 'amount_sum': 45.0}]

Acero’s building blocks#

The node types listed in the Acero user’s guide, checked 2026-10-11.

KindNodesWhat they do
Sourcestable_source, record_batch_source, record_batch_reader_source, exec_batch_source, array_vector_source, source, scanFeed batches in from memory, a reader, a custom async generator, or files through Arrow Datasets
Computefilter, project, aggregate, pivot_longerDrop rows, compute columns, summarize (grouped or not), turn columns into rows
Arrangementhashjoin, asofjoin, union, order_by, fetchJoin, merge, reorder or slice streams
Sinkssink, table_sink, consuming_sink, writeUsually chosen for you by DeclarationToTable, DeclarationToReader or DeclarationToStatus

See it in a real stack

No architecture uses Apache Arrow Acero yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2021–2026

Arrow is a shared in-memory columnar format, and its C++ library grew compute functions and a datasets layer for reading collections of files. The compute functions work on data that is already in memory; scanning many files, joining and grouping needed a way to stream batches through a series of operations. Arrow 5.0.0 in July 2021 added an ExecNode hierarchy, wrapped grouped aggregation in a node and rebuilt dataset scans as graphs of these nodes. Arrow 6.0.0 in October 2021 shipped what its release post called a basic in-memory query engine, with filter, project, sort, equality joins and aggregations, and R users got grouped aggregation and joins in dplyr on top of it.

In May 2022 the developers reached consensus on a name, Acero, and from Arrow 9.0.0 (August 2022) the docs call it “Acero: A C++ streaming execution engine”. Arrow 12.0.0 in May 2023 moved Acero out of the core libarrow into its own shared library, so the core build could shrink, added the order_by, fetch and pivot_longer nodes, and gave Python a pyarrow.acero module. Its documentation sets out what it is not: not a database, not an optimizer, not distributed, not a tool for data scientists to use directly.

Why it’s built this way

No parser, planner or optimizer
The docs call creating the best plan very hard and argue the optimizer should live outside Acero, composed through standards such as Substrait so any backend can use it.
Streams of batches, not tables
Working on one batch at a time removes the need for multi-batch containers and avoids awkward cases such as table columns chunked at different sizes.
Substrait as the preferred input
Substrait producers offer friendlier APIs than hand-built node graphs, and a plan in a standard format can move to another Substrait-consuming engine.
A separate library from Arrow 12
Splitting libarrow_acero out of libarrow lets projects that only need the format and compute functions ship smaller builds.

Timeline

  1. 2021

    Arrow 5.0.0 (July) adds the ExecNode hierarchy and runs dataset scans as node graphs

  2. 2021

    Arrow 6.0.0 (October) ships a basic query engine, used by R’s dplyr bindings

  3. 2022

    Developers settle on the name Acero in May; Arrow 9.0.0 docs use it in August

  4. 2023

    Arrow 12.0.0 (May) splits out libarrow_acero and adds the pyarrow.acero module

  5. 2026

    Arrow 26.0.0 (9 October) still ships Acero, documented as experimental

Sources8 · facts checked 11 October 2026

Improve this page on GitHub