Databricks

A managed lakehouse platform built on Apache Spark and Delta Lake, with Unity Catalog for governance, on AWS, Azure or Google Cloud.

What it is: Lakehouses,Data warehouses,Query engines
Type: Commercial product
Made by: Databricks, Inc.
License: Proprietary
Facts checked: 5 October 2026

What it is

Databricks runs Spark clusters, SQL warehouses, notebooks and jobs as a managed service, while the data stays in cloud object storage such as S3, Azure Data Lake Storage or Google Cloud Storage, as Delta Lake tables by default. Photon, a vectorised engine written in C++, replaces Spark’s JVM-based execution for SQL and DataFrame work without code changes, and is on by default in SQL warehouses and serverless compute.

Unity Catalog governs tables, files, ML models and other assets with one permission model and automatic lineage. Iceberg tables in Unity Catalog, including managed Iceberg tables (spec v1 to v3), are reachable by outside engines through its Iceberg REST catalog API for reads and writes. Iceberg tables owned by another catalog can be queried but are read-only in Databricks. Unity Catalog also exists as an open-source project under LF AI & Data, Apache 2.0 licensed.

Why teams pick it

One platform for ETL, SQL analytics, streaming and machine learning on open Delta and Iceberg files
Photon and serverless SQL warehouses make warehouse-style BI work on lake tables
Unity Catalog: fine-grained permissions and lineage across workspaces
Runs on all three major clouds

Watch out for

Databricks bills in Databricks Units (DBUs), and on classic compute your cloud provider bills separately for the machines, so there are two bills to track
The formats are open, but much of the speed and convenience (Photon, predictive optimisation, serverless) is Databricks-only
Iceberg tables managed by other catalogs are read-only in Databricks
A large platform to administer: workspaces, compute policies, networking and permissions

Alternatives

What all lakehouses have in common →

Amazon S3 TablesAWSA bucket type for Apache Iceberg tables: S3 stores them, acts as their catalog, and compacts and cleans them up for you.

Apache AmoroIncubatingOpen sourceAn incubating Apache “lakehouse management system”, started at NetEase as Arctic, that watches Iceberg tables and keeps them fast on its own: it compacts small files, rewrites deletes and expires old snapshots and files in the background.

DremioCommercialA SQL lakehouse engine built on Apache Arrow, with its own Iceberg catalog and “reflections” that speed up queries without anyone changing them.

SpiralCommercialA closed-source storage and query engine for multimodal AI data (video, images, sensor traces and tables) that keeps everything as Vortex files in your own object storage, versions every change and streams results to GPUs; access is by request.

Tiger LakeBetaCommercialTiger Cloud’s built-in Iceberg connector, in public beta: it copies a Postgres table or TimescaleDB hypertable into Apache Iceberg, then keeps it current from the write-ahead log.

How it fits with other technologies

What Databricks uses

What uses Databricks

Delta and Iceberg tables side by side in Unity Catalog (Databricks SQL) · sql

-- names are catalog.schema.table
CREATE TABLE main.sales.orders (
  order_id BIGINT, customer_id BIGINT, amount DECIMAL(10,2), ts TIMESTAMP
) CLUSTER BY (ts);

CREATE TABLE main.sales.refunds (refund_id BIGINT, order_id BIGINT, amount DECIMAL(10,2))
USING ICEBERG;

GRANT SELECT ON TABLE main.sales.orders TO `analysts`;

OPTIMIZE main.sales.orders;
DESCRIBE HISTORY main.sales.orders;

See it in a real stack

No architecture uses Databricks yet. Drop it into the playground and wire it up yourself.Open playground →

Worth exploring

Article· August 2021What Is a Data Lakehouse and Answers to Other Frequently Asked QuestionsMichael Armbrust, Bharath Gowda, Reynold Xin, Matei Zaharia and Ali Ghodsi, DatabricksDatabricks answers ten questions: how a lakehouse differs from a data lake and a warehouse, how it compares on cost and speed, what governance it supports, and how it relates to a data mesh. Databricks sells a lakehouse, so read it as the case for one.

All resources →

Benchmarks

Benchmark · ClickHouse Inc.ClickBenchHow fast analytical databases run 43 SQL queries over one 100-million-row table of web analytics events, on the same cloud machines.Includes Databricks · 7 caveats to read first

How to read a benchmark →

Where it came from2013–2026

By 2013, Apache Spark was taking off, but big data clusters were hard to set up and manage, and getting value from data meant gluing together a hodgepodge of tools. That year the team that started the Spark research project at UC Berkeley founded Databricks, with Ion Stoica as CEO. In June 2014 they unveiled Databricks Cloud: hosted Spark clusters with notebooks, dashboards and a job launcher, so a team could go from raw data to results without running infrastructure. It became generally available in June 2015, launching clusters on Amazon EC2.

Customers kept their data in cheap object stores like S3, but those stores had no transactions: a crashed update job left a table half-written, and queries over tens of thousands of objects were slow. In the service’s first years (2014–2016), around half of Databricks’ support escalations came from problems like these. The answer was Delta Lake, a transaction log on top of Parquet files, open-sourced in April 2019. In a 2021 CIDR paper, Armbrust, Ghodsi, Xin and Zaharia argued that this “Lakehouse” would replace the two-tier setup of a data lake plus a separate warehouse, which left warehouse data stale and copied twice.

Since then Databricks has added a C++ engine (Photon), governance (Unity Catalog, open-sourced in 2024) and AI tooling, and bought Tabular, the company of Apache Iceberg’s creators, to bridge the two main table formats.

Why it’s built this way

Keep data in the customer’s own object storage: Object stores let companies scale storage and compute separately: store a petabyte but run a cluster for a few hours. Databricks brought the compute to that data instead of loading it into its own storage.
A transaction log on top of Parquet files: Object stores are key-value stores with expensive listing and weak consistency. Delta Lake records every change in a log, compacted into Parquet, which gives ACID commits, time travel and fast metadata lookups without a database server.
Open formats instead of a proprietary warehouse: The lakehouse bet was that one copy of data in open files could serve BI, data science and ML. Tools like TensorFlow and PyTorch need to read large datasets directly, which is slow through a warehouse’s ODBC or JDBC interface.
Photon: a native C++ engine instead of the JVM: Faster storage and data skipping had made queries CPU-bound, and tuning JVM-generated code hit limits, including garbage collection trouble on heaps over 64 GB. A vectorised C++ engine gave control over memory and SIMD.

Timeline

2013
Founded by the creators of Apache Spark
2014
Databricks Cloud unveiled in June: hosted Spark with notebooks
2015
Platform becomes generally available in June, on Amazon EC2
2019
Delta Lake open-sourced in April
2021
Lakehouse paper at CIDR argues for one platform on open formats
2024
Acquires Tabular (Iceberg’s creators); open-sources Unity Catalog
2026
Announces in July it is raising a strategic round at a $188 billion valuation

Sources7 · facts checked 5 October 2026

Improve this page on GitHub