Cloud Parts

DuckDB

An in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.

What it is
Query engines,Relational databases
Type
Open source
Made by
DuckDB Foundation and DuckLabs
License
MIT
Facts checked
5 October 2026
Website Source code

What it is

DuckDB runs inside your process, the way SQLite does, but it’s built for analytics: columnar storage and a vectorised engine that works on batches of values in parallel threads, spilling to disk when a query outgrows memory. It has no external dependencies and installs as a library for Python, R, Java or Node.js, or as a single command-line binary. A database is one file, or you can skip that and query files where they are: SELECT ... FROM 's3://bucket/*.parquet'.

Extensions add formats and storage. httpfs reads and writes S3 and HTTP; iceberg reads Iceberg tables and writes them through a REST catalog; delta reads and writes Delta Lake using the Delta Kernel library; postgres queries a live PostgreSQL database; and ducklake attaches DuckLake catalogs. The current release is 1.5.6, and the 1.4 line (1.4.5) is the long-term support release.

Why teams pick it

  • Zero setup: pip install duckdb and you have a fast columnar engine
  • Reads Parquet, CSV, JSON, Iceberg and Delta Lake in place, locally or on S3, reading only the columns and row groups it needs
  • Larger-than-memory queries spill to disk, so one laptop or VM goes a long way
  • MIT licensed, with the IP held by the non-profit DuckDB Foundation

Watch out for

  • One machine only: it scales up, not out
  • Only one process can write to a database file at a time; it isn’t a shared server for many users
  • Iceberg writes need a REST catalog; reading a table by its path is read-only
  • Built for analytics, not many small transactional writes; use PostgreSQL or SQLite for app data

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

What uses DuckDB

Query Parquet and Iceberg on S3, then write results back · sql

CREATE SECRET (TYPE s3, PROVIDER credential_chain);

SELECT country, count(*) AS orders
FROM 's3://acme-raw/orders/2026/*.parquet'
GROUP BY country ORDER BY orders DESC LIMIT 10;

INSTALL iceberg;
SELECT count(*) FROM iceberg_scan('s3://acme-lake/warehouse/sales/orders');

COPY (FROM 's3://acme-raw/orders/2026/*.parquet' WHERE amount > 100)
  TO 's3://acme-lake/big-orders' (FORMAT parquet, PARTITION_BY (country));

See it in a real stack

Small lakehouse on DuckLake →
Live model
What you’re watchingA nightly load writes new rows: DuckLake commits the snapshot to PostgreSQL and the data to S3 as Parquet. Analysts’ queries look up the current files in the catalog and read them from S3.

Worth exploring

WebsiteAwesome DuckDBDavid Gasquez and contributors, on GitHubA community-curated list of more than 400 DuckDB links: client APIs, core and community extensions, BI and ETL tools built on DuckDB (grouped by commercial, open core and open source), WebAssembly and SQL clients, plus talks, podcasts, blog posts and books. Use it to find what already exists around DuckDB; entries arrive by pull request with one-line descriptions, so judge each project yourself.
All resources →

Benchmarks

Benchmark · ClickHouse Inc.ClickBenchHow fast analytical databases run 43 SQL queries over one 100-million-row table of web analytics events, on the same cloud machines.Includes DuckDB · 7 caveats to read first
How to read a benchmark →
Where it came from2018–2026

By the late 2010s, a data scientist with a few gigabytes in R or pandas had two poor options. A database server meant installing, configuring and loading data through a slow client protocol. SQLite ran inside the program with no setup, but it was built for transactions, with a row-by-row engine over B-trees, so analytical queries ran very slowly. The dataframe tools themselves offered relational-style operators but no query optimiser.

Mark Raasveldt and Hannes Mühleisen, database researchers at CWI in Amsterdam, had already built MonetDBLite, an embedded version of the MonetDB analytical database. It was popular, used from the Dutch central bank to the New Zealand police, but adapting a system that wasn’t built for embedding caused problems that were hard to fix. In 2018 they started again from scratch: a purpose-built analytical database that runs inside another program, as SQLite does. They presented DuckDB at SIGMOD 2019 in Amsterdam, MIT licensed from the start.

In 2021 the founders spun off a company from CWI, DuckDB Labs, to employ the core engineering team, and set up the non-profit DuckDB Foundation to hold most of the IP. DuckDB 1.0 arrived in June 2024 with a file format that later versions promise to keep reading. In May 2026 DuckDB Labs renamed itself DuckLabs. In August 2026 DuckLabs announced it would join AWS as a new subsidiary, expected to take effect in early September; DuckDB, DuckLake and Quack stay MIT licensed under the stewardship of the DuckDB Foundation.

Why it’s built this way

Run inside the host process, like SQLite
An embedded library needs no server, and because the database and application share memory, tables can move between DuckDB and pandas or R without being copied over a socket.
A columnar, vectorised engine instead of SQLite’s row engine
Analytical queries touch a few columns across many rows. Processing columns in batches of values uses the CPU far better than handling one row at a time.
No external dependencies, and never crash the host
MonetDBLite taught them that depending on external libraries caused trouble and that an embedded database that crashes takes the user’s program down with it. Queries that run out of memory must abort cleanly.

Timeline

  1. 2018

    First code written at CWI in Amsterdam

  2. 2019

    First release; demo paper at SIGMOD in Amsterdam

  3. 2021

    DuckDB Labs spins off from CWI; the DuckDB Foundation is set up

  4. 2024

    DuckDB 1.0 in June, with a stable storage format

  5. 2025

    1.4.0 ships as a long-term support release; every other version will be LTS

  6. 2026

    DuckLake 1.0 in April; DuckDB Labs becomes DuckLabs in May; DuckLabs announces in August it will join AWS

Sources17 · facts checked 5 October 2026

Improve this page on GitHub