Data lake, lakehouse or warehouse?
| Dimension | Data lakes | Lakehouses | Data warehouses |
|---|---|---|---|
| In one line | Raw files in cheap storage, given a structure when read | Tables made of open files, with transactions | A managed database built for analytics |
| Where the data lives | Your object storage, in any format: JSON, CSV, Parquet, images | Your object storage, as Parquet files tracked by a table format | The warehouse’s own managed storage, in its own format |
| Schema | Applied on read; nothing checks data on the way in | Enforced on write; columns can change without rewriting files | Enforced on write |
| Updates and deletes | No transactions: rewrite the affected files yourself, and hope no reader catches you halfway | Atomic commits; MERGE, UPDATE and DELETE through the engine | Built in |
| Who can read it | Any engine that reads the files, each with its own table definitions unless they share a catalog | Any engine that speaks the table format, through a shared catalog | Mainly the warehouse’s own engine |
| Data it suits | Anything, including images, audio and documents | Tables, with other files kept alongside them | Tables and semi-structured JSON |
| What you operate | Storage is easy; keeping the lake findable and trustworthy is the real work | Catalog, compaction, snapshot expiry and permissions, unless a platform manages them | Little: tuning, compaction and access control are managed |
| Cost at scale | Object-storage prices; engines cost money only while they run | Object-storage prices; engines cost money only while they run | Storage and compute priced by the vendor |
| Reach for it for | Cheap raw storage, and files that aren’t tables | One copy of data shared by BI, ML and streaming | Quick BI wins with a small team |
Rule of thumb
They stack more than they compete. A lakehouse is a data lake with a table format on top, and most warehouses now read and write Iceberg tables in it. Ask where the authoritative copy lives, whether writes are transactions, and who else can read it without a copy job.
Sources
- Lakehouse: A New Generation of Open Platforms (CIDR 2021)
- Thusoo et al.: Hive, a warehousing solution over a Map-Reduce framework (VLDB 2009)
- AWS: What is a data lake?
- Athena docs: CREATE TABLE (external tables over files in S3)
- Iceberg table spec: snapshots, atomic commits, schema evolution
- Iceberg docs: Spark writes (MERGE INTO, UPDATE, DELETE FROM)
- Iceberg docs: Vendors (engines that read and write Iceberg tables)
- Iceberg docs: Maintenance (compaction, snapshot expiry)
- Snowflake docs: Apache Iceberg tables
- Snowflake docs: Semi-structured data
- BigQuery docs: Apache Iceberg managed tables
- BigQuery docs: Working with JSON data
- Databricks docs: Databricks SQL
- Amazon S3 pricing
- Amazon Athena pricing (pay per query)