Cloud Parts

Data lakes

Raw files of any kind in cheap storage, given a structure only when someone reads them.

3 technologies on this page ↓
Live model
What you’re watchingApps and database dumps drop files into the raw zone as they are, with no checks. A clean-up job rewrites them as Parquet in the curated zone. Analysts’ SQL goes through table definitions that only point at folders, so nothing stops a half-written file or a renamed column from reaching the answer.

How it works

A data lake is a large pool of files in cheap storage: today almost always object storage such as S3, Azure Data Lake Storage or Google Cloud Storage, and before that Hadoop’s HDFS. Data lands as it is, whether JSON logs, CSV exports, Parquet, images or PDFs, and nothing checks it on the way in. Structure is applied when someone reads it (“schema on read”). For SQL, a catalog such as the Hive Metastore or AWS Glue Data Catalog maps a folder to a table and its subfolders (dt=2026-09-30/) to partitions, and engines such as Spark, Trino or Athena scan the files. Most teams split the lake into zones, such as raw, cleaned and curated, with jobs that rewrite one zone into the next. What a plain lake lacks is transactions. A table is whatever files sit in its folder right now, so concurrent writers, half-finished jobs and row-level deletes are your problem. Add a table format on top and you have a lakehouse.

Think of it as

A storage unit you can fill with anything, without filling in a form. Moving in is easy. Whether you find the right box a year later depends on how tidy everyone was.

Reach for it when

  • Keeping raw data cheaply, before you know every question you’ll ask of it
  • Files that don’t fit in tables: images, audio, documents, model training sets
  • A landing zone that feeds a warehouse or lakehouse

Think twice when

  • Tables that need updates, deletes or several writers at once; add a table format and use a lakehouse
  • Business users who expect clean, documented tables; without ownership and a catalog, a lake turns into a swamp

A real example

A mobile game writes every client event as gzipped JSON under s3://acme-lake/raw/events/dt=…/. A nightly Glue job turns each day into Parquet in the curated zone, and analysts query it from Athena, paying per terabyte scanned. When a player asks for their data to be deleted, though, someone has to find and rewrite every file that holds their events.

Technologies

Specific products, and what each does differently.

Also used for data lakes

Products of other kinds that are often used this way, grouped by what they are.

What each provider calls it

AWS
S3, with Glue Data Catalog, Lake Formation permissions, and Athena or EMR to query
Google Cloud
Cloud Storage, with BigQuery external tables and Managed Service for Apache Spark (formerly Dataproc)
Azure
Azure Data Lake Storage (Blob Storage with a hierarchical namespace), with Azure Databricks or Microsoft Fabric
Open source & others
S3-compatible storage such as MinIO, Ceph or SeaweedFS, or Hadoop’s HDFS, plus a Hive Metastore and Spark, Trino or DuckDB

In code · sql

-- Athena: the table only describes files that are already in S3
CREATE EXTERNAL TABLE events (
  player_id string,
  event     string,
  ts        timestamp
)
PARTITIONED BY (dt string)
STORED AS PARQUET
LOCATION 's3://acme-lake/curated/events/';

-- register the dt=... folders written since the table was created
MSCK REPAIR TABLE events;

SELECT event, count(*) AS n
FROM events
WHERE dt = '2026-09-30'
GROUP BY event;

Often confused with

LakehousesData warehousesObject storageFile formatsQuery enginesETL & ELT pipelinesData lake, lakehouse or warehouse?
Where it came from2006–2020

In the 2000s, analytics meant a data warehouse. Data was cleaned and modelled before it was loaded, and storage cost enough that anything without a clear use was thrown away. Hadoop, which split off from the Nutch search project in 2006, changed the economics: HDFS spread files across racks of cheap servers, and MapReduce processed them where they sat. Facebook built Hive on top so analysts could use SQL, with each table stored as a directory in HDFS and each partition as a subdirectory.

In October 2010 James Dixon, then CTO of Pentaho, contrasted the data mart, “a store of bottled water – cleansed and packaged and structured for easy consumption”, with a “data lake”: “a large body of water in a more natural state” that users could “examine, dive in, or take samples” from. Hadoop vendors took up the term, and critics followed. In July 2014 Gartner warned of a “data lake fallacy”, since lakes “lack semantic consistency and governed metadata”, and in December 2014 Michael Stonebraker argued that without data curation the lake is really a “data swamp”.

From around 2015, cloud object stores took over from HDFS, so storage and compute could scale separately and engines such as Presto and Athena queried files in place. Object stores weren’t file systems, though: until December 2020, a job listing an S3 folder could miss files that had just been written. The missing piece was transactions across many files. Table formats such as Iceberg, Delta Lake and Hudi added them, and a lake with a table format on top is what is now called a lakehouse.

Why it’s built this way

Schema on read
Modelling data before loading it was slow, and it threw away whatever nobody had asked about yet. Storing data raw keeps every option open, at the cost of making each reader work out what the data means.
Open files, not a database
Any engine can read CSV, JSON and later Parquet, so a team could add Spark next to Hive, or Presto next to both, without moving data.
Folders as tables and partitions
Hive mapped a table to a directory and its partitions to subdirectories such as ds=20090101, so queries could skip whole folders. That needed no new storage layer, but a table could only ever be as consistent as a directory listing.

Timeline

  1. 2006

    Hadoop splits off from Nutch, bringing cheap storage on commodity servers

  2. 2009

    Facebook describes Hive: SQL tables mapped onto HDFS directories

  3. 2010

    James Dixon coins “data lake” in a blog post

  4. 2014

    Gartner warns of the “data lake fallacy”; Stonebraker calls lakes “data swamps”

  5. 2016

    Amazon Athena launches, running SQL directly on files in S3

  6. 2020

    S3 becomes strongly consistent, so a listing now shows files just written

Sources13

Improve this page on GitHub