In-memory columnar

Apache Arrow fixes how a column sits in memory, byte for byte: a validity bitmap that marks nulls, a buffer of values and, for strings and lists, an offsets buffer. Every language uses the same layout, so a C++ library, a Rust engine and a Python program can pass each other pointers to the same buffers instead of serialising and parsing.

It works alongside the on-disk formats rather than against them. Parquet is encoded and compressed to be small on disk, Arrow is laid out to be computed on, and most tools read Parquet straight into Arrow. When Arrow data does go to disk, its IPC file format (also called Feather v2) keeps the in-memory bytes, so a reader can memory-map the file without decoding it.