Columnar files

Apache Parquet and Apache ORC cut a table into large row groups (ORC calls them stripes) and, inside each, store every column as a separate compressed chunk. A footer holds the schema plus min/max statistics for every chunk, so an engine reads only the columns a query names and skips row groups whose statistics rule them out.

That suits analytics, which reads a few columns across many rows, and fits single-row lookups poorly. Newer formats aim at that gap: Vortex and Lance both claim random access about 100 times faster than Parquet, Lance by dropping row groups altogether. Apache TsFile tunes the layout for sensor data instead, grouping each device’s series into chunks that carry their own timestamps and statistics.

Table formats build on these files rather than replacing them. Apache Iceberg tracks Parquet, ORC or Avro data files as one table, and Delta Lake does the same with Parquet.