Cloud Parts

Query engines

Compute that runs SQL over data it doesn’t own, and scales separately from the storage.

31 technologies on this page ↓
Live model
What you’re watchingTwo engines share one copy of the data. Analysts’ queries go to a coordinator that hands them to a pool of workers, which grows with demand and shrinks when it’s quiet. A nightly job spins up its own Spark cluster, reads the same files and shuts down again. Neither engine owns the storage.

How it works

A query engine parses your SQL, asks a catalog which files hold the table, and splits the work into tasks that many workers run in parallel. Each worker reads its share of Parquet files straight from object storage, then results are shuffled, aggregated and returned through a coordinator. Because the data stays in open files, several engines can share it: an interactive SQL engine for dashboards, Spark for heavy batch and ML jobs, DuckDB on a laptop. Each one scales up, or shuts down, without moving any data.

Think of it as

Researchers in a public library. You can bring in one or a hundred and they all read the same shelves. When they go home, the books stay put.

Reach for it when

  • Interactive SQL over a data lake without loading it into a warehouse first
  • Several teams or tools that need the same tables through different engines
  • Spiky analytics, where paying per query or per second beats an always-on cluster

Think twice when

  • Sub-second lookups for an app; use a database or a cache
  • Small data: DuckDB or a single Postgres is simpler than a distributed cluster

A real example

A retailer keeps three years of clickstream as Iceberg tables in S3. Analysts run Athena queries that each scan a few GB and pay per TB scanned. A nightly Spark job on EMR builds features from the same tables, and an engineer debugs one day’s partition with DuckDB on a laptop.

Technologies

Specific products, and what each does differently.

Distributed engines and MPP databases

Clusters that split one query across many machines.

TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.DremioCommercialA SQL lakehouse engine built on Apache Arrow, with its own Iceberg catalog and “reflections” that speed up queries without anyone changing them.Mainly LakehousesDatabricksCommercialA managed lakehouse platform built on Apache Spark and Delta Lake, with Unity Catalog for governance, on AWS, Azure or Google Cloud.Mainly LakehousesStarRocksOpen sourceAn open-source MPP analytical database with a vectorized engine that serves fast joins and real-time updates on its own tables, and also queries Iceberg, Hudi, Delta Lake and Hive tables in place.Mainly Real-time analytics databasesApache DorisOpen sourceAn open-source MPP analytical database, born at Baidu as Palo, that serves sub-second queries on fresh, updatable data through the MySQL protocol and also queries Iceberg, Hudi, Paimon and Hive tables in place.Mainly Real-time analytics databases

Managed and serverless engines

Someone else runs the machines; you pay per query, scan or use.

Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Google BigQueryGoogle CloudGoogle Cloud’s serverless data warehouse: no clusters, and you pay per TiB scanned or per slot-hour.Mainly Data warehousesMotherDuckCommercialA serverless cloud data warehouse built on DuckDB, where one query can run partly on your laptop and partly in the cloud.Mainly Data warehousesHotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.SpiralCommercialA closed-source storage and query engine for multimodal AI data (video, images, sensor traces and tables) that keeps everything as Vortex files in your own object storage, versions every change and streams results to GPUs; access is by request.Mainly Lakehouses

Single-node and embedded engines

One process, often a library inside your program, sometimes wrapped as a small server.

DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.PivotOpen sourceA young open-source analytics server in Rust that speaks the PostgreSQL protocol and queries Iceberg and Delta Lake tables straight from object storage.Mainly Real-time analytics databases

Building blocks for engine builders

Parsers, planners and execution libraries that other engines are made of.

Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometVeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.

What each provider calls it

AWS
Athena, Redshift Spectrum, EMR (Spark, Trino), Glue
Google Cloud
BigQuery (incl. BigLake tables), Managed Service for Apache Spark (formerly Dataproc)
Azure
Microsoft Fabric (SQL endpoint, Spark), Azure Databricks, Synapse serverless SQL
Open source & others
Trino, Presto, Apache Spark, Apache Hive, Apache Impala, DuckDB, Polars (a DataFrame library with a streaming engine), StarRocks, ClickHouse, Apache DataFusion (a library to build engines on), chDB (ClickHouse as a library), GizmoSQL (DuckDB served over Arrow Flight SQL), Spice.ai OSS (federates sources and caches them locally, built on DataFusion); also Dremio (now part of SAP), Starburst

See it in a real stack

Architectures that use query engines

Small lakehouse on DuckLakeDuckDB writes and queries tables whose catalog lives in PostgreSQL and whose data sits as Parquet on S3.DuckDBDuckLakePostgreSQLAmazon S3Open lakehouse on IcebergKafka streams events into Spark, which writes Iceberg tables on S3 that Trino queries for dashboards.Apache KafkaApache SparkApache IcebergAmazon S3Trino

In code · python

import duckdb

con = duckdb.connect()
con.sql("CREATE SECRET (TYPE s3, PROVIDER credential_chain)")

# DuckDB reads only the columns and row groups this query needs
con.sql("""
    SELECT date_trunc('day', ts) AS day, count(*) AS views
    FROM read_parquet('s3://lake/clicks/2026/09/*.parquet')
    GROUP BY day ORDER BY day
""").show()

Benchmarks

Benchmark · ClickHouse Inc.ClickBenchHow fast analytical databases run 43 SQL queries over one 100-million-row table of web analytics events, on the same cloud machines.32 systems in this catalog · 7 caveats to read first
How to read a benchmark →

Often confused with

Data warehousesLakehousesTable formatsAutoscaling
Where it came from2009–2018

For most of database history, the SQL engine and the storage were one product: to query data, you loaded it into the database, in the database’s own format. In the late 2000s, companies such as Facebook began keeping huge datasets as plain files in Hadoop’s HDFS, because traditional warehouses had become prohibitively expensive at that size. But querying those files meant writing low-level MapReduce programs that were hard to maintain and reuse.

Facebook’s Hive, described at VLDB 2009, put SQL on top of those files. A table was just a directory, a catalog (the Hive Metastore) recorded schemas and formats, and each query compiled into MapReduce jobs; Facebook’s Hive warehouse already held over 700 TB. Hive was slow, so a wave of engines followed that read the same files without MapReduce: Google’s Dremel paper (2010) showed interactive SQL on data in place, Cloudera announced Impala and Facebook built Presto in 2012, and Spark SQL followed. AWS made the idea serverless with Athena in 2016, billing per byte scanned.

Sharing files between engines had a weakness: no transactions, so a half-finished write could corrupt a table. Table formats such as Apache Iceberg, created at Netflix in 2017, added that, which is why several engines can now safely work on the same tables.

Why it’s built this way

A catalog that maps tables to files
Once storage was just files, an engine needed somewhere to learn which files make up a table and how to read them. Hive’s Metastore filled that role, and later catalogs such as Glue and Iceberg REST catalogs followed the same idea.
Pipelined, in-memory execution instead of MapReduce
Hive’s MapReduce jobs wrote every stage to disk. Presto and Dremel-style engines stream results between stages in memory, trading some fault tolerance for interactive speed.
Open columnar file formats
Dremel’s encoding for nested columns was adopted by Apache Parquet in 2013, so any engine could read only the columns a query needs, without one vendor owning the format.

Timeline

  1. 2009

    Facebook describes Hive, SQL over files in Hadoop

  2. 2010

    Google’s Dremel paper shows interactive SQL on data in place

  3. 2012

    Cloudera announces Impala; Facebook starts Presto

  4. 2013

    Presto open-sourced; Twitter and Cloudera announce Parquet

  5. 2016

    Amazon Athena offers serverless SQL over S3, billed per byte scanned

  6. 2018

    Netflix’s Iceberg table format enters the Apache Incubator

Sources27

Improve this page on GitHub