Cloud Parts

Apache Parquet

The standard columnar file format for analytics: each column stored and compressed on its own, with statistics that let engines skip data.

What it is
File formats (columnar files)
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
5 October 2026
Website Source code

What it is

A Parquet file splits its rows into row groups. Inside each row group, every column is stored as its own column chunk, broken into pages. Values of one type sit next to each other, so dictionary and run-length encodings plus a codec such as Snappy or ZSTD shrink them well. The footer comes last, so a writer can produce the file in one pass, and it holds the schema and min/max statistics for every column chunk.

A reader fetches the footer first, then reads only the column chunks a query names and skips row groups whose statistics rule out a match. Nested data (structs, lists, maps) is flattened into plain columns with the record shredding algorithm from Google’s Dremel paper. Twitter and Cloudera started Parquet in 2013. It’s now an Apache project and the default data file format under table formats such as Iceberg and Delta Lake.

Why teams pick it

  • Read by practically every engine and library, including Spark, Trino, DuckDB, BigQuery, Athena and pandas
  • Queries read only the columns and row groups they need, often a small fraction of the file
  • Compresses well, because each column gets its own encoding and compression codec
  • The schema, including nested types, travels inside the file

Watch out for

  • A file can’t be edited in place: changing one row means rewriting the file, which is why table formats exist
  • Built for batches; writing one event at a time is slow and produces tiny files
  • Thousands of small files scan slowly; the format’s own guidance is large row groups of 512 MB to 1 GB
  • A folder of Parquet files isn’t a table: no transactions, no shared schema, no record of which files belong

Alternatives

What all columnar files have in common →
Apache ORCOpen sourceA columnar file format born in Hive: typed columns in large stripes, built-in indexes every 10,000 rows, and the storage behind Hive’s ACID tables.Apache TsFileOpen sourceA columnar file format for device time series, with per-series indexes and time-series encodings; the storage format of Apache IoTDB and its own Apache project since 2023.PuffinOpen sourceApache Iceberg’s file format for the data that doesn’t fit in a manifest: a container of typed binary blobs, such as distinct-value sketches and deletion vectors, described by a JSON footer.VortexOpen sourceA columnar file format and toolkit that keeps data compressed with modern encodings, reads single rows fast and runs filters directly on the compressed data, pitched as a faster successor to Parquet.

How it fits with other technologies

What uses Apache Parquet

Look inside a Parquet file with PyArrow · python

import pyarrow.parquet as pq

pf = pq.ParquetFile("readings.parquet")
print(pf.schema_arrow)
print(pf.metadata.num_row_groups, "row groups")

# the min/max statistics an engine uses to skip row groups
chunk = pf.metadata.row_group(0).column(0)
print(chunk.path_in_schema, chunk.statistics.min, chunk.statistics.max)

# read one column from one row group; nothing else is touched
temps = pf.read_row_group(0, columns=["temp_c"])

See it in a real stack

No architecture uses Apache Parquet yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2010–2026

Google’s 2010 Dremel paper showed that storing nested records column by column let it aggregate trillion-row tables in seconds. Hadoop users wanted the same for their files on HDFS, and by early 2013 several columnar formats were competing, including Doug Cutting’s Trevni and ORC, which Hortonworks and Facebook built to speed up Hive. One Hadoop user replied to the next announcement: “Do we really need 3 columnar formats?”

That announcement, on 12 March 2013, introduced Parquet as a joint project between Twitter and Cloudera engineers. Julien Le Dem started it at Twitter together with Cloudera’s Impala team, and Criteo engineers signed up to build the Hive support. The goal was to make compressed columnar storage “available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model, or programming language.” Twitter began converting its major data sources to save on storage and deserialization. Le Dem later wrote that the name was picked to evoke “the bottom layer of a database with an interesting layout.”

Version 1.0 shipped in July 2013. Parquet entered the Apache Incubator in May 2014 and became a top-level project in April 2015. It went on to become the default data file under almost every lake engine and table format.

Why it’s built this way

Dremel’s repetition and definition levels for nested data
Hadoop data was often nested objects (the first release converted Thrift objects directly). Storing each leaf field as a column plus small level numbers keeps the nesting without flattening field names.
Compression per column, separate from encoding
Each column picks its own codec, and encodings are kept apart from compression so engines can work on encoded values without paying to decode them.
A neutral format with no favourite engine
The founders said they were “not interested in playing favorites” and shipped building blocks for Pig, Hive, Impala, Thrift and Avro, so no single framework owned the format.

Timeline

  1. 2010

    Google’s Dremel paper describes columnar storage for nested records

  2. 2013

    Twitter and Cloudera announce Parquet in March; 1.0 ships in July

  3. 2014

    Enters the Apache Incubator in May

  4. 2015

    Graduates to a top-level Apache project in April

  5. 2025

    The spec adds a variant type and geometry and geography types

  6. 2026

    Format 2.14.0 adds ALP, a compact encoding for floating-point columns

Sources16 · facts checked 5 October 2026

Improve this page on GitHub