Cloud Parts

Ibis

A Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.

What it is
Query engines
Type
Open source
Made by
Ibis project (core team sponsored by Voltron Data)
License
Apache 2.0
Facts checked
9 October 2026
Website Source code

What it is

Ibis looks like a dataframe library: you filter, group_by, agg, join and order_by tables in Python. But nothing runs when you write those lines. Ibis builds a lazy expression, and only an action such as to_pandas(), to_pyarrow() or to_polars() sends it to a backend. For most backends Ibis compiles the expression to that engine’s SQL dialect with SQLGlot; the rest are DataFrame backends that get the engine’s own native calls. ibis.to_sql(expr) shows the SQL, and table.sql("...") mixes hand-written SQL into a pipeline.

The backend is chosen by the connection, such as ibis.duckdb.connect() or ibis.bigquery.connect(...), so the idea is to develop against a local file and deploy against the warehouse by changing one line. DuckDB is the default and recommended backend for local data. The list includes DataFusion, Druid, Flink, Impala, PySpark, Athena, BigQuery, ClickHouse, Databricks, DuckDB, Exasol, Materialize, MySQL, Oracle, Polars, PostgreSQL, RisingWave, SingleStoreDB, SQL Server, SQLite, Snowflake and Trino. Because the target is SQL, Ibis follows SQL semantics: tables have no index and no guaranteed row order. Google’s BigQuery DataFrames, a pandas-style API, is built on Ibis.

Why teams pick it

  • One API for more than 20 engines: prototype on DuckDB or Polars with a sample, then point the same code at BigQuery, Snowflake or Trino
  • Interactive mode for exploring data step by step, with the same expressions reused in pipeline code
  • Composes with SQL instead of hiding it: inspect the generated SQL with ibis.to_sql, or start from a SQL query and continue in Python
  • Reads CSV and Parquet files directly through the backend, e.g. con.read_parquet(...) on DuckDB, Polars or DataFusion

Watch out for

  • Not a drop-in pandas replacement: no index, no positional access and no guaranteed row order, so pandas code needs rewriting
  • Not every operation works on every backend; Ibis publishes an operation support matrix, and portable code sticks to the common subset
  • Breaking changes across major versions: 9.0 (May 2024) replaced SQLAlchemy with SQLGlot internally, and 10.0 (February 2025) removed the pandas and Dask backends
  • Its own docs say Voltron Data is the primary sponsor and employs most of the core team, so the project depends heavily on one company

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

One lazy expression: printed as PostgreSQL SQL, executed on DuckDB · python

import ibis
from ibis import _

con = ibis.duckdb.connect()  # in-memory DuckDB; swap for ibis.postgres.connect(...) etc.
orders = con.create_table("orders", ibis.memtable({
    "country": ["DE", "DE", "FR", "US", "US", "US"],
    "amount":  [120, 80, 50, 300, 20, 90],
}))

revenue = (
    orders.filter(_.amount > 30)          # lazy: nothing runs yet
    .group_by("country")
    .agg(revenue=_.amount.sum(), n=_.count())
    .order_by(ibis.desc("revenue"))
)

print(ibis.to_sql(revenue, dialect="postgres"))
# SELECT * FROM (SELECT "t1"."country", SUM("t1"."amount") AS "revenue", COUNT(*) AS "n"
#   FROM (SELECT * FROM "memory"."main"."orders" AS "t0" WHERE "t0"."amount" > 30) AS "t1"
#   GROUP BY 1) AS "t2" ORDER BY "t2"."revenue" DESC NULLS LAST

print(revenue.to_pandas())  # runs on DuckDB
#   country  revenue  n
# 0      US      390  2
# 1      DE      200  2
# 2      FR       50  1

See it in a real stack

No architecture uses Ibis yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2015–2026

Wes McKinney, the creator of pandas, started Ibis in 2015 as a pandas-like interface to Apache Impala. The first release on PyPI was 0.3.0 in July 2015.

The project grew from one engine to many. For years it had no default backend, because nobody on the team was comfortable making pandas, the only in-memory engine at the time, the default. DuckDB changed that: it handles larger-than-memory CSV, Parquet and JSON files, and became the recommended backend for local data. Under the stewardship of Phillip Cloud and a team employed by Voltron Data, which McKinney co-founded, Ibis 9.0 in May 2024 finished “the big refactor” from SQLAlchemy to SQLGlot, and Ibis 10.0 in February 2025 removed the pandas and Dask backends. Version 12.0 followed in February 2026.

Why it’s built this way

Lazy expressions, not eager dataframes
A deferred expression can be compiled and shipped to the engine that holds the data. pandas computes each step as you go, which is why the Ibis team said making it behave in a deferred way was hard and eventually dropped it as a backend.
Compile to SQL
Most data engines speak SQL and many have no Python API. Generating each engine’s dialect lets one Python API reach all of them, and since 9.0 SQLGlot does the dialect work.
SQL semantics, not pandas semantics
Ibis has no index and makes no promise about row order, because, as its pandas guide puts it, SQL “makes no guarantees about row order”. Ibis tables follow the same rules as the database tables they stand for.
DuckDB as the default
DuckDB offered “the uber-convenience of SQLite but for analytic use cases” and handles larger-than-memory CSV, Parquet and JSON files, so it became the recommended way to work with local data.

Timeline

  1. 2015

    Wes McKinney starts Ibis for Apache Impala; 0.3.0 on PyPI in July

  2. 2019

    Ibis 1.0 released in March

  3. 2024

    Ibis 9.0 in May completes the move from SQLAlchemy to SQLGlot

  4. 2025

    Ibis 10.0 in February removes the pandas and Dask backends

  5. 2026

    Ibis 12.0 released in February

Sources12 · facts checked 9 October 2026

Improve this page on GitHub