Cloud Parts

File formats

How rows and columns are laid out inside a data file, which decides how much a query has to read.

17 technologies on this page ↓
Live model
What you’re watchingThe same question asked of two copies of the data. Against CSV, the query has to read every column of every row before it can answer. Against Parquet, it reads the footer, goes straight to the one column it needs and answers far sooner. The other 39 columns are never read.

How it works

Row formats such as CSV, JSON lines and Avro write one whole record after another. That’s ideal for appending events, but a query has to read every field of every row. Columnar formats such as Apache Parquet and ORC split rows into row groups (ORC calls them stripes) and store each column separately inside them, compressed with encodings that suit the values. A footer holds the schema plus min/max statistics for every column chunk, so an engine reads only the columns a query names and skips row groups that can’t match.

Think of it as

A filing cabinet with one drawer per field. To total every invoice amount you open the “amount” drawer, instead of pulling out every folder and reading each page.

Reach for it when

  • Parquet for anything a warehouse, Spark, Trino or DuckDB will scan
  • Avro or JSON lines for event streams and logs written one record at a time
  • CSV only for small hand-offs to spreadsheets and partners

Think twice when

  • Single-row lookups or frequent in-place updates; that’s a database
  • Thousands of tiny Parquet files: per-file overhead dominates, so compact them into files of a few hundred MB (Iceberg’s default target is 512 MB)

A real example

Sensors write readings with 40 fields each. As gzipped CSV, a month is 1.2 TB, and “average temperature per day” reads all of it. Rewritten as Parquet with ZSTD compression it shrinks to about 150 GB, and the same query reads only the ts and temp_c columns: around 8 GB.

Kinds of file formats

Specific products, and what each does differently.

Columnar files →

Files on disk that store each column on its own, compressed, so a query reads only the columns it names and skips blocks that can’t match.

Apache ORCOpen sourceA columnar file format born in Hive: typed columns in large stripes, built-in indexes every 10,000 rows, and the storage behind Hive’s ACID tables.Apache ParquetOpen sourceThe standard columnar file format for analytics: each column stored and compressed on its own, with statistics that let engines skip data.Apache TsFileOpen sourceA columnar file format for device time series, with per-series indexes and time-series encodings; the storage format of Apache IoTDB and its own Apache project since 2023.PuffinOpen sourceApache Iceberg’s file format for the data that doesn’t fit in a manifest: a container of typed binary blobs, such as distinct-value sketches and deletion vectors, described by a JSON footer.VortexOpen sourceA columnar file format and toolkit that keeps data compressed with modern encodings, reads single rows fast and runs filters directly on the compressed data, pitched as a faster successor to Parquet.LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.Mainly Table formats

In-memory columnar →

A shared layout for tables in RAM, so libraries and engines in different languages can hand each other data without copying or converting it.

Apache ArrowOpen sourceA standard way to lay out columnar data in memory, so libraries, engines and languages can share tables without copying or converting them.

Array formats →

Storage for dense and sparse multi-dimensional arrays, such as images, climate grids, genomic variants and LiDAR points, read by slicing coordinates.

TileDBOpen sourceAn embeddable C++ storage engine and open format for dense and sparse multi-dimensional arrays, on local disk or cloud object storage.

Binary serialization →

Compact, schema-driven encodings for one record or message at a time: the bytes between services and inside event streams.

Apache AvroOpen sourceA compact, row-based binary format with a JSON schema, widely used for Kafka messages and record files.CBOROpen sourceConcise Binary Object Representation: a binary format with JSON’s data model plus byte strings and tags, designed for tiny decoders and readable without a schema.Cap’n ProtoOpen sourceA binary format laid out like an in-memory struct, so reading needs no parse step, plus an RPC system built on object references and promise pipelining.Mainly Remote procedure callsProtocol BuffersOpen sourceGoogle’s schema-first binary encoding: messages defined in `.proto` files, fields identified by number, code generated for each language.Mainly Remote procedure calls

What each provider calls it

AWS
Open formats: Athena, Redshift Spectrum and Glue read Parquet, ORC, Avro, CSV and JSON
Google Cloud
BigQuery loads and external tables over Parquet, ORC, Avro, CSV and JSON
Azure
Fabric (over OneLake) and Synapse serverless SQL query Parquet, CSV and JSON files
Open source & others
Apache Parquet, Apache ORC, Lance (fast row lookups for AI data), Vortex (compressed encodings you can filter without decompressing; LF AI & Data), TileDB (dense and sparse multi-dimensional arrays, MIT), Apache Avro, Apache Arrow (in memory), CSV, JSON and JSON Lines; for messages: Protocol Buffers, Cap’n Proto; for configuration: YAML, TOML and XML; for describing existing binary formats: Kaitai Struct

In code · python

import pyarrow.csv as csv
import pyarrow.parquet as pq

table = csv.read_csv("readings.csv")
pq.write_table(table, "readings.parquet", compression="zstd")

# read two columns; the other 38 are never touched
temps = pq.read_table("readings.parquet", columns=["ts", "temp_c"])
print(temps.num_rows, temps.nbytes)

Often confused with

Table formatsQuery enginesObject storageData warehousesFile format or table format?
Where it came from2005–2025

For decades, data moved between systems as text. CSV was used to swap data between spreadsheet programs for so long that when RFC 4180 finally wrote it down in 2005, it noted the format had “never been formally documented”. Row files like CSV, and later JSON lines, are easy to write, but a query that needs one column still has to read every byte of every row.

Databases had already found the alternative. In 2005 Michael Stonebraker and colleagues at MIT, Brown, Brandeis and UMass Boston published C-Store, a database that stores data by column to make reads fast, and Google’s 2010 Dremel paper showed how to lay out nested records in columns and aggregate trillion-row tables in seconds. Hadoop users wanted the same for files. Doug Cutting’s Avro (2009) gave Hadoop compact row files with a schema, and in early 2013, a month apart, Hortonworks and Facebook announced ORC to speed up Hive, and Twitter and Cloudera announced Parquet.

Parquet won the broadest support and became the default data file under table formats such as Iceberg and Delta Lake. Apache Arrow (2016) applied the same columnar layout in memory, so engines and languages can hand data to each other without converting it. Newer formats such as Lance revisit the trade-offs for AI data, where fetching single rows matters as much as scanning.

Why it’s built this way

Store each column together, not each row
Analytical queries read a few columns of many rows. Keeping a column’s values together means a query reads only what it needs, and similar values next to each other compress far better.
Put the schema inside the file
CSV carries no types, so every reader has to guess or be told. Avro and Parquet embed their schema, so any tool can open a file without outside definitions.
Shred nested records into flat columns
Real data has lists and structs. Dremel’s repetition and definition levels let nested records be stored as plain columns and rebuilt on read, and Parquet adopted the technique.

Timeline

  1. 2005

    C-Store paper makes the case for column stores; RFC 4180 documents CSV

  2. 2009

    Avro brings schema-described binary row files to Hadoop

  3. 2010

    Google’s Dremel paper shows columnar storage for nested data

  4. 2013

    ORC (February) and Parquet (March) announced for Hadoop

  5. 2016

    Apache Arrow standardises columnar data in memory

  6. 2025

    Parquet adds variant and geospatial types; a Lance paper tackles random access

Sources26

Improve this page on GitHub