File format or table format?
| Dimension | File formats | Table formats |
|---|---|---|
| Describes | How one file stores its rows and columns | Which files make up a table, and its history |
| Examples | Parquet, ORC, Avro, CSV | Iceberg, Delta Lake, Hudi |
| Transactions | None; a file is written once | Atomic commits across many files |
| Updates and deletes | Rewrite the files yourself | MERGE, UPDATE and DELETE through the engine |
| Schema changes | Each file carries its own schema | Tracked centrally; columns change without rewrites |
| Time travel | No | Yes, until old snapshots are expired |
Rule of thumb
They stack rather than compete. A table format is metadata that points at files in a file format, almost always Parquet, so you choose one of each.
Sources
- Apache Parquet docs: File format
- Apache ORC specification v1
- Apache Avro specification (schema stored in each container file)
- RFC 4180: Common Format and MIME Type for CSV Files
- Iceberg table spec: snapshots, atomic commits, schema and partition evolution
- Iceberg docs: Spark writes (MERGE INTO, UPDATE, DELETE FROM)
- Iceberg docs: Evolution (schema changes are metadata changes)
- Iceberg docs: Spark queries (time travel)
- Iceberg docs: Maintenance (expire snapshots)
- Iceberg docs: Configuration (write.format.default is parquet)
- Delta transaction log protocol (Parquet data files)
- Hudi docs: Table and query types