cloudparts

Apache Drill

A distributed SQL engine, inspired by Google’s Dremel, that queries JSON, Parquet, CSV files and NoSQL stores in place without defining a schema first.

Apache · Top-level project
What it is
Query engines
Type
Open source
Made by
Apache Software Foundation
Foundation
The Apache Software Foundation, top-level project, joined August 2012
License
Apache 2.0
Facts checked
24 October 2026
Website Source code

What it is

Drill runs a service called a Drillbit on each node, ideally next to the data, and uses ZooKeeper to keep track of which Drillbits are alive. A client sends SQL from the Drill shell, the web UI, a JDBC driver or the REST API to any Drillbit, which becomes the Foreman for that query: it parses and optimizes the SQL with Apache Calcite, turns the physical plan into major fragments (phases) and splits each into minor fragments, units of work that run as threads on Drillbits close to the data. Drill needs no central catalog. Storage plugins describe each source: file systems (local, HDFS, S3, Azure Blob Storage, SFTP), Hive, HBase, MongoDB, Cassandra, Elasticsearch, Kafka, Druid, Splunk, JDBC databases, HTTP APIs and Google Sheets.

Its main idea is schema on read: Drill discovers the schema while it reads each batch of records, so you can point SQL at a directory of JSON or Parquet files and query nested fields straight away, with no CREATE TABLE. Directory names become the columns dir0, dir1 and so on, which it uses to skip partitions. One query can join a file in S3 with a Hive table and a MongoDB collection. The project is still maintained by a small group of committers: 1.22.0, released in June 2025, dropped Java 8, and work since then has added materialized views, an Accumulo plugin and Java 25 support, though no newer release had shipped by October 2026.

Why teams pick it

  • Queries self-describing data such as JSON and Parquet in place, with no schema definitions or loading step
  • One SQL query can join across files, Hive, HBase, MongoDB, Kafka and JDBC databases through storage plugins
  • A hierarchical columnar data model handles nested and changing data, with functions such as FLATTEN for nested arrays
  • An embedded JDBC driver, a web UI and a REST API, so tools can use it like a database (the ODBC driver comes from a vendor, not the project)

Watch out for

  • Activity is low: few regular contributors, and the last release (1.22.0) is from June 2025
  • Schema discovery at read time can fail or slow down on inconsistent files, such as a JSON field that changes type
  • Its Hadoop-era design, with long-running Drillbits and ZooKeeper, is more work to run than serverless engines like Athena or embedded ones like DuckDB

Specs

Compare with alternatives →
Data model
Hierarchical columnar model; schema discovered while reading
Query language
SQL, with functions for nested data such as FLATTEN
Client protocol
JDBC, REST API and web UI; ODBC from a vendor
How it scales
A Drillbit per node; each query split into fragments that run near the data
Storage format
None of its own: JSON, Parquet and CSV files, Hive, HBase, MongoDB, Kafka, JDBC and more
Written in
Java
Ways to run it
Self-hosted Drillbits, coordinated through ZooKeeper

Alternatives

What all query engines have in common →Compare side by side →
TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.+1 DuckFlightDremioCommercialA SQL lakehouse engine built on Apache Arrow, with its own Iceberg catalog and “reflections” that speed up queries without anyone changing them.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.+1 Apache GlutenApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.

How it fits with other technologies

Query files in place: no table definitions, and directory levels become columns · sql

-- a sample JSON file shipped on Drill's classpath
SELECT * FROM cp.`employee.json` LIMIT 5;

-- Parquet files under multilevel/parquet/<year>/<quarter>/
SELECT * FROM dfs.`multilevel/parquet/199[4-5]/*`;

-- dir0 is the first directory level below the queried path
USE dfs.logs;
SELECT * FROM logs WHERE dir0 = '2013' LIMIT 3;

See it in a real stack

No architecture uses Apache Drill yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2010–2026

In 2010 Google described Dremel, which ran ad-hoc aggregations over trillion-row tables of read-only nested data in seconds. Hadoop’s MapReduce gave high throughput but not the low latency people wanted for exploring data. In 2012 engineers at MapR Technologies, with contributors from Concurrent and Drawn to Scale, proposed Drill to the Apache Software Foundation as an open-source take on Dremel: interactive analysis of large, nested datasets. It entered the Apache Incubator in September 2012 and became a top-level project in November 2014.

Drill 1.0, the first production-ready release, came out in May 2015. That year Tomer Shiran, one of Drill’s first committers, and Jacques Nadeau, who ran MapR’s distributed systems team, founded Dremio. Drill itself kept going as a community project; in recent years a small group of committers, with Charles Givre the most active, has fixed bugs and added storage plugins, format readers and, in 2026, materialized views.

Why it’s built this way

Discover the schema while reading
Self-describing formats like JSON and Parquet already carry their structure, so requiring table definitions first would slow down exploration and break on data that changes shape.
No central metadata store
Each storage plugin supplies its own metadata, so Drill can query many Hive metastores, HBase tables and file systems at once without copying their catalogs.
A peer-to-peer cluster of identical Drillbits
Any Drillbit can take a query and act as its Foreman, so there is no master node, and Drillbits can sit on the data nodes for locality.

Timeline

  1. 2010

    Google publishes the Dremel paper

  2. 2012

    Drill enters the Apache Incubator in September, proposed by MapR engineers

  3. 2014

    Graduates to a top-level Apache project in November

  4. 2015

    Drill 1.0, the first production-ready release, in May

  5. 2025

    Drill 1.22.0 drops Java 8 and the MapR plugin (June)

  6. 2026

    Materialized views and Java 25 support land on the main branch

Sources12 · facts checked 24 October 2026

Improve this page on GitHub