Cloud Parts

Polars

A DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.

What it is
Query engines
Type
Open source
Made by
Polars (the company), started by Ritchie Vink
License
MIT (Polars Cloud, the company’s managed platform, is a separate commercial service)
Facts checked
6 October 2026
Website Source code

What it is

Polars looks like pandas: you load a table and chain operations on it. Underneath it treats each chain as a query. With the lazy API (scan_parquet, scan_csv and friends) nothing runs until you call collect(). Polars first builds a plan, then its optimiser pushes filters and column selection down into the file scan, reads only the slice it needs, caches repeated subplans and picks a join order. The engine is written in Rust, runs vectorised and multi-threaded, and holds data in the Apache Arrow columnar format, so it can hand tables to PyArrow, DuckDB and other Arrow tools without copying. SQL works too: Polars translates it into the same expressions instead of running a separate SQL engine.

Since Polars 2.0 (6 October 2026) collect() uses the streaming engine by default. It processes the data in batches rather than all at once, so datasets larger than RAM fit, and operations it can’t stream fall back to the in-memory engine. Spilling to disk is on by default too, starting at about 80% of RAM, though in 2.0 only some operations (sorts, window functions, many expressions) can spill yet. An optional GPU engine built with NVIDIA’s cuDF runs the same queries on a GPU. The library stays on one machine; Polars Cloud, a commercial platform from the company behind Polars, runs the same queries across a cluster on AWS or your own Kubernetes.

Why teams pick it

  • Uses every core, an optimiser and columnar Arrow memory, where older DataFrame libraries ran single-threaded and step by step
  • The streaming engine handles data larger than RAM on a laptop or a single server, with no cluster to run
  • Zero-copy Arrow interchange with PyArrow, DuckDB and other Arrow tools, plus readers for Parquet, CSV, JSON, Delta Lake, Iceberg and cloud storage
  • A strict, consistent API that catches schema and type errors early instead of silently coercing data
  • MIT licensed, with a funded company paying for core development

Watch out for

  • Not a drop-in pandas replacement: there’s no row index and the API is different, so existing code has to be rewritten
  • The open-source library runs on one machine; distributed execution comes through Polars Cloud, the company’s commercial platform
  • 2.0 just changed defaults: the streaming engine doesn’t keep row order for joins and group-bys unless you ask (maintain_order), and joins and group-bys can’t spill to disk yet
  • The headline TPC-H and TPC-DS comparisons with DuckDB and DataFusion were run and published by the Polars team

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

A lazy query: planned first, then streamed · python

import polars as pl

q = (
    pl.scan_parquet("s3://acme-lake/orders/*.parquet")
    .filter(pl.col("status") == "shipped")
    .group_by("customer_id")
    .agg(
        pl.col("amount").sum().alias("total"),
        pl.len().alias("n_orders"),
    )
    .sort("total", descending=True)
)

print(q.explain())   # the optimised plan: filter and columns pushed into the scan
top = q.collect()    # 2.0: runs on the streaming engine by default

One query, several engines#

The same LazyFrame can run on different engines. You pick with collect(engine=...), or Polars picks for you.

Streaming

default since 2.0

Processes data in batches using morsel-driven parallelism, so it needs less memory and can handle data larger than RAM. Falls back to the in-memory engine for operations it can’t stream yet.

In-memory

engine="in-memory"

The original engine: processes the whole dataset in memory at once. Still used for operations the streaming engine doesn’t cover, and selectable as the default.

GPU

open beta

Built with NVIDIA RAPIDS cuDF. Install polars[gpu] and call collect(engine="gpu"); a query it can’t run falls back to the CPU. Python only.

Distributed

Polars Cloud

The company’s platform splits a query across workers that each run the open-source streaming engine. Generally available on AWS since September 2025, and on your own Kubernetes since June 2026.

SQL on DataFrames#

SQL is translated into Polars expressions and runs on the same engine, so it can mix freely with DataFrame code.

python
import polars as pl

orders = pl.scan_parquet("orders.parquet")
customers = pl.read_csv("customers.csv")

ctx = pl.SQLContext(orders=orders, customers=customers)
top = ctx.execute("""
    SELECT c.country, sum(o.amount) AS revenue
    FROM orders o JOIN customers c USING (customer_id)
    GROUP BY c.country ORDER BY revenue DESC LIMIT 10
""").collect()

See it in a real stack

No architecture uses Polars yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2020–2026

Ritchie Vink made the first commit on 23 June 2020, stuck at home in the first year of the pandemic. It began as a pet project to learn about query engines, Apache Arrow and Rust, because the Rust DataFrame tools of the time didn’t satisfy him. By February 2021 it was near the top of H2O.ai’s database-like benchmark, behind only R’s data.table among DataFrame libraries, and he introduced it with a post titled “I wrote one of the fastest DataFrame libraries”.

His view was that a DataFrame is a front end on a query engine. Single-machine DataFrame libraries were mostly single-threaded, kept everything in RAM and ignored decades of database research, while laptops kept getting more cores. Polars was written from scratch to use them, with a query optimiser so that idiomatic code still runs fast. In August 2023 Vink and Chiel Peters started a company around it with a seed round of about $4 million led by Bain Capital Ventures, promising that Polars would stay MIT licensed.

Python Polars 1.0 (July 2024) declared the in-memory engine production ready and announced a redesigned streaming engine. A GPU engine with NVIDIA followed in September 2024. In September 2025 the company launched Polars Cloud on AWS with a distributed engine in beta, then raised an €18 million Series A led by Accel. Polars 2.0, released on 6 October 2026, made the streaming engine the default.

Why it’s built this way

A query engine behind a DataFrame API
Eager DataFrames run each step as written. Making the lazy API central lets an optimiser see the whole query, push filters and column choices into the file scan, and skip work, so users can write readable code without hand-tuning it.
Arrow’s memory format, in its own Rust implementation
Columnar Arrow memory suits vectorised execution and lets Polars swap data with other tools without copying. Polars keeps its own implementation, polars-arrow, forked from Jorge Leitão’s arrow2, so it controls its API, memory and execution end to end.
Strict and fail fast
Vink wanted users to get “the hangover up front”: errors when a query is planned, not twenty minutes into a pipeline. Polars 2.0 went further, refusing lossy type coercions and mismatched concatenations that earlier versions allowed.
A streaming engine built on morsels and Rust async
To process data larger than memory, the redesigned engine combines morsel-driven parallelism (small chunks of rows spread across cores) with Rust’s async state machines, giving a hybrid push and pull design.

Timeline

  1. 2020

    Ritchie Vink makes the first commit (23 June)

  2. 2021

    “I wrote one of the fastest DataFrame libraries” introduces Polars

  3. 2023

    Vink and Chiel Peters found the company with a ~$4M seed

  4. 2024

    Python Polars 1.0 (July); GPU engine beta with NVIDIA (September)

  5. 2025

    Polars Cloud generally available on AWS; €18M Series A

  6. 2026

    Distributed engine on Kubernetes (June); Polars 2.0 makes streaming the default (October)

Sources18 · facts checked 6 October 2026

Improve this page on GitHub