Cloud Parts

Lakehouses

Warehouse-style tables on open files in object storage, shared by every engine.

21 technologies on this page ↓
Live model
What you’re watchingIngest, dashboards and model training all use the same tables. Every engine goes through one catalog to find the current snapshot, then reads the same open files in object storage. There is no copy of the data per tool.

How it works

A lakehouse keeps all its data in object storage as open files, usually Parquet, organized into tables by a table format such as Iceberg or Delta Lake. A catalog registers the tables and controls who may read or change them. On top, any number of engines work on the same tables: SQL for dashboards, Spark or Python for data science, streaming jobs for fresh data. You get a warehouse’s transactions and SQL without copying data into a proprietary store, and a lake’s low cost and openness without a swamp of untracked files.

Think of it as

A shared kitchen with labelled, inventoried shelves. Every cook can bring their own tools, and the inventory means nobody grabs a half-unpacked crate.

Reach for it when

  • One copy of data serving BI, ML and streaming, instead of a copy per tool
  • Large or fast-growing data, where warehouse storage gets expensive
  • Avoiding lock-in: any engine that speaks the table format can read the tables

Think twice when

  • A small team whose data a managed warehouse already handles well
  • You can’t staff the extra moving parts: catalog, compaction, snapshot expiry and permissions

A real example

An energy company lands smart-meter readings as Iceberg tables in S3 Tables. Operators’ dashboards query them through Athena, data scientists train forecasting models in Spark on the same tables, and finance reads them from Snowflake. Nobody runs a copy job.

Technologies

Specific products, and what each does differently.

Amazon S3 TablesAWSA bucket type for Apache Iceberg tables: S3 stores them, acts as their catalog, and compacts and cleans them up for you.Apache AmoroIncubatingOpen sourceAn incubating Apache “lakehouse management system”, started at NetEase as Arctic, that watches Iceberg tables and keeps them fast on its own: it compacts small files, rewrites deletes and expires old snapshots and files in the background.DatabricksCommercialA managed lakehouse platform built on Apache Spark and Delta Lake, with Unity Catalog for governance, on AWS, Azure or Google Cloud.DremioCommercialA SQL lakehouse engine built on Apache Arrow, with its own Iceberg catalog and “reflections” that speed up queries without anyone changing them.SpiralCommercialA closed-source storage and query engine for multimodal AI data (video, images, sensor traces and tables) that keeps everything as Vortex files in your own object storage, versions every change and streams results to GPUs; access is by request.Tiger LakeBetaCommercialTiger Cloud’s built-in Iceberg connector, in public beta: it copies a Postgres table or TimescaleDB hypertable into Apache Iceberg, then keeps it current from the write-ahead log.

Also used for lakehouses

Products of other kinds that are often used this way, grouped by what they are.

Table formats →

Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.

Lakehouse catalogs →

Apache PolarisOpen sourceAn open-source Iceberg REST catalog: one server that every engine asks where a table’s current metadata is, with role-based access control and short-lived storage credentials per table.Cloudflare Basin CatalogRenamedCommercialCloudflare’s managed Apache Iceberg REST catalog built into R2 buckets, called R2 Data Catalog until it became generally available as Basin Catalog on 1 October 2026, which also compacts tables and expires old snapshots for you.LakekeeperOpen sourceAn open-source Apache Iceberg REST catalog written in Rust: a stateless server, backed by PostgreSQL, that tells every engine where each table’s current metadata is, checks permissions, and hands out short-lived storage credentials.Project NessieOpen sourceAn open-source catalog for Apache Iceberg with Git-like branches, tags and commits that span every table at once, so a set of changes to many tables can be tested on a branch and then merged in one atomic step.Unity CatalogCommercialDatabricks’ catalog and governance layer for tables, files, functions and models, with managed and external tables, an Iceberg REST endpoint and credential vending; a separate Apache 2.0 open-source server lives at LF AI & Data.

What each provider calls it

AWS
S3 Tables, Glue Data Catalog, Lake Formation, with Athena, Redshift and EMR
Google Cloud
Lakehouse (formerly BigLake) and BigQuery, with Spark on Managed Service for Apache Spark (formerly Dataproc)
Azure
Microsoft Fabric (OneLake), Azure Databricks
Open source & others
Iceberg or Delta Lake + Trino or Spark + a catalog such as Polaris or Unity Catalog; also Databricks, Snowflake, Dremio (now part of SAP)

In code · python

from pyiceberg.catalog import load_catalog

# one catalog, shared by every engine that reads these tables
catalog = load_catalog("lake", type="rest", uri="https://catalog.example.com", warehouse="analytics")

meters = catalog.load_table("energy.meter_readings")
df = meters.scan(
    row_filter="reading_date >= '2026-09-01'",
    selected_fields=("meter_id", "kwh"),
).to_pandas()
print(df.groupby("meter_id").kwh.sum().head())

Worth exploring

Article· September 2025Understanding Apache FlussJack VanlightlyA long, detailed walk through Fluss internals, reverse-engineered from the code: log tables, primary-key tables that emit changelogs, and tiering to lakehouse tables. The author works at Confluent, which runs Flink but not Fluss, and says so up front.
Article· August 2021What Is a Data Lakehouse and Answers to Other Frequently Asked QuestionsMichael Armbrust, Bharath Gowda, Reynold Xin, Matei Zaharia and Ali Ghodsi, DatabricksDatabricks answers ten questions: how a lakehouse differs from a data lake and a warehouse, how it compares on cost and speed, what governance it supports, and how it relates to a data mesh. Databricks sells a lakehouse, so read it as the case for one.
All resources →

Often confused with

Data lakesData warehousesTable formatsLakehouse catalogsQuery enginesObject storageETL & ELT pipelinesData lake, lakehouse or warehouse?
Where it came from2010–2025

Data warehouses gave businesses reliable SQL over cleaned data, but they coupled compute and storage in expensive appliances and couldn’t hold video, audio or free text. From around 2010, companies poured raw data into cheap Hadoop storage instead; Pentaho’s James Dixon called it a “data lake”, in contrast to the tidy data mart. From 2015, cloud object stores such as S3, ADLS and GCS began replacing HDFS. Most large companies ended up with two tiers: a lake for raw data and machine learning, and a warehouse such as Redshift or Snowflake fed with a curated copy for BI.

Two tiers meant ETL twice, warehouse data that was days stale, and two systems to keep consistent. Table formats built from 2016 onward (Hudi at Uber, Iceberg at Netflix, Delta Lake at Databricks) added transactions and snapshots to lake files. In January 2020 Databricks described the result as a lakehouse, “a new, open architecture that combines the best elements of data lakes and data warehouses”, and in a CIDR 2021 paper Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia argued that the warehouse architecture “will wither in the coming years”.

Warehouses and clouds converged on the idea rather than fighting it. Engines such as Snowflake, BigQuery and Athena now read open tables, AWS launched managed Iceberg tables as S3 Tables in December 2024, and open catalogs such as Apache Polaris (a top-level Apache project since February 2026) let many engines share one set of tables.

Why it’s built this way

One copy of data, in open files
Copying lake data into a warehouse added cost, ETL failures and staleness. Keeping a single copy in open formats like Parquet lets warehouses, Spark and ML tools all read the same files.
A transactional metadata layer over the files
Plain folders of files have no atomic updates or versions. Table formats add a log or snapshot tree, which brings ACID transactions, time travel and schema enforcement to the lake.
Storage separate from compute
Warehouse appliances forced companies to pay for peak load. With data in object storage, any number of engines can start, scale and stop independently of the data.

Timeline

  1. 2010

    James Dixon coins “data lake” for raw data kept in Hadoop

  2. 2015

    Cloud object stores start replacing HDFS as the home of data lakes

  3. 2017

    Delta Lake launches at Databricks; Iceberg starts at Netflix

  4. 2020

    Databricks describes the lakehouse in a January blog post

  5. 2021

    CIDR paper argues lakehouses will replace the warehouse architecture

  6. 2024

    AWS launches S3 Tables; Apache Polaris catalog enters the incubator

  7. 2025

    DuckLake proposes keeping lakehouse metadata in a SQL database

Sources14

Improve this page on GitHub