Apache Arrow

A standard way to lay out columnar data in memory, so libraries, engines and languages can share tables without copying or converting them.

What it is: File formats (in-memory columnar)
Type: Open source
Made by: Apache Software Foundation
License: Apache 2.0
Facts checked: 5 October 2026

What it is

Arrow defines how a table sits in memory, byte for byte. Each column is an array made of a few contiguous buffers: a validity bitmap that marks nulls, the values themselves and, for strings and lists, an offsets buffer that says where each value starts. Columns that share a schema are grouped into record batches. Because the layout is the same in every language, a C++ library, a Rust engine and a Python program can hand each other pointers to the same buffers instead of serialising and parsing. Inside one process this goes through the C Data Interface: two small C structs, with no Arrow library needed on either side.

Between processes, the IPC format writes the same buffers out: a stream format for pipes and sockets, and a file format with a footer for random access. A Feather v2 file is exactly an Arrow IPC file, and since the bytes on disk match the bytes in memory, a reader can memory-map it without decoding anything. Arrow Flight sends record batches over gRPC, Flight SQL builds a database protocol on top of Flight, and ADBC is a client API like JDBC or ODBC that returns columns instead of rows, with drivers for PostgreSQL, Snowflake, DuckDB, BigQuery and others. Parquet is the storage partner: Parquet is encoded and compressed to be small on disk, Arrow is laid out to be computed on, and most tools read Parquet straight into Arrow. Official libraries cover C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, Rust and Swift; the current release is 25.0.1.

Why teams pick it

Two systems that both speak Arrow exchange tables without converting them, in one process or over the network
Columns sit in contiguous, aligned buffers, so vectorised (SIMD) code runs on them directly
Libraries in a dozen languages, integration-tested against each other so they read each other’s data
IPC files can be memory-mapped and used with no deserialisation step
Flight and ADBC move query results as columns, instead of row-by-row JDBC or ODBC

Watch out for

It isn’t an archive format: Parquet is built for space efficiency and is usually the better choice for data at rest
Zero-copy only works when both sides already use Arrow’s layout; data from a row-based system or plain Python objects is still converted once
Arrow is plumbing, not a database: there is no catalog, storage management or SQL server, so you pair it with an engine such as DuckDB, DataFusion or Spark
Feature coverage differs by language; check the implementation status table before relying on Flight or a newer data type

How it fits with other technologies

What Apache Arrow uses

What uses Apache Arrow

Parquet into Arrow, query it with DuckDB, save an IPC file · python

import duckdb
import pandas as pd
import pyarrow as pa
import pyarrow.feather as feather
import pyarrow.parquet as pq

# Parquet on disk -> Arrow in memory, two columns only
readings = pq.read_table("readings.parquet", columns=["sensor", "temp_c"])

# DuckDB scans the Arrow table by its variable name and returns Arrow
peaks = duckdb.sql("SELECT sensor, max(temp_c) AS peak FROM readings GROUP BY sensor").to_arrow_table()

# pandas columns backed by Arrow arrays
df = peaks.to_pandas(types_mapper=pd.ArrowDtype)

# an uncompressed IPC (Feather v2) file can be memory-mapped and read without decoding
feather.write_feather(peaks, "peaks.arrow", compression="uncompressed")
with pa.memory_map("peaks.arrow") as source:
    same = pa.ipc.open_file(source).read_all()

Arrow, Arrow IPC files or Parquet?#

	Arrow in memory	Arrow IPC / Feather file	Parquet file
Where it lives	RAM, shared between libraries in one process	A pipe, socket or file	A file on disk or object storage
Built for	Computing on columns	Moving Arrow data between processes	Storing data compactly for years
Reading it	Nothing to read: pass a pointer	Memory-map it; no decoding when uncompressed	Decode and decompress into memory, usually into Arrow
Size	Uncompressed	Optional LZ4 or ZSTD compression; Feather defaults to LZ4	Encoded and compressed per column: smallest
Use it for	Handing tables between engines and languages	Caches, scratch files, fast hand-offs between tools	Data lakes, table formats, archives

The Arrow family#

Everything below shares the same columnar layout, so data passes between the pieces without conversion.

Columnar format

spec

The byte-level layout of arrays, buffers and record batches. Declared stable, with compatibility guarantees, in 1.0 (2020).

C Data Interface

in-process

Two C structs that let libraries in one process share Arrow data with no build-time dependency on each other.

IPC format

between processes

A stream format and a file format with a footer for random access. Feather v2 is the file format under its old name.

Flight

network

An RPC framework on gRPC that streams record batches with DoGet, DoPut and DoExchange, and can split a result across several endpoints read in parallel.

Flight SQL

database protocol

SQL queries, prepared statements and catalog lookups over Flight. A database implements the server side and reuses a generic client.

ADBC

client API

Arrow Database Connectivity: a JDBC/ODBC-style API that returns Arrow, with drivers for PostgreSQL, SQLite, Snowflake, BigQuery, DuckDB, Flight SQL and more.

See it in a real stack

No architecture uses Apache Arrow yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2015–2023

By 2015, almost every analytics system had its own way of holding columns in memory. Moving a table from a JVM engine such as Spark or Impala into pandas meant serialising every value and parsing it again; the projects involved estimated that 70 to 80% of CPU time in many workloads went on this kind of conversion. Each project also rebuilt the same columnar code from scratch.

Jacques Nadeau, co-founder of Dremio, had been working on a standard in-memory columnar format for the same reason. Wes McKinney, the creator of pandas, then at Cloudera, wanted Python to work with systems like Impala and Spark without expensive copies. Together they recruited developers from Calcite, Drill, Impala, Kudu, Parquet, Spark and other projects, and on 17 February 2016 the Apache Software Foundation announced Arrow as a top-level project straight away, skipping the incubator. A month later McKinney and Hadley Wickham released Feather, a file format for passing data frames between Python and R that grew out of the Arrow discussions.

Version 1.0 in July 2020 declared the columnar format stable. Flight, Flight SQL and ADBC then carried the same layout onto the network and into database drivers, and pandas 2.0 (April 2023) let data frames keep their columns as Arrow arrays.

Why it’s built this way

Specify the bytes, not just an API: If every implementation lays out buffers identically, a C++ library and a Python program can use the same memory. An API alone would still need conversion at every boundary.
Contiguous, aligned buffers per column: The spec recommends 8- or 64-byte alignment and padding so vectorised loops can use SIMD instructions without extra checks.
A neutral Apache project from day one: McKinney described the ASF as a “neutral ‘safe space’ free from software vendor affiliation”, so competing engines could adopt the format without depending on one company.
Leave long-term storage to Parquet: Arrow is laid out for computation and Parquet for space efficiency. The project treats them as complements, and invests in fast Parquet-to-Arrow readers instead of a rival file format.

Timeline

2015
Nadeau and McKinney gather developers from a dozen projects around a shared format
2016
Top-level Apache project in February; Feather in March; 0.1.0 in October
2019
DataFusion donated in February; Arrow Flight introduced in October
2020
C Data Interface in May; 1.0 declares the format stable in July
2022
Flight SQL introduced in February
2023
ADBC in January; pandas 2.0 adds Arrow-backed columns in April

Sources24 · facts checked 5 October 2026

Improve this page on GitHub