Cloud Parts

Velox

Meta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

What it is
Query engines
Type
Open source
Made by
Meta
License
Apache 2.0
Facts checked
11 October 2026
Website Source code

What it is

Every query engine has the same parts: a language front end, an optimizer, a runtime that spreads tasks over machines, and an execution engine that does the actual work on one machine. Velox is only that last part, as a library. It has no SQL parser, no dataframe API and no optimizer. A host engine such as Presto or Spark plans the query and hands Velox a plan fragment; Velox runs it with the CPU and memory of the local node. Inside are a type system with nested types, columnar Vectors, a vectorized expression evaluator, Presto- and Spark-compatible function packages, operators (scans, writes, filters, projections, hash, merge and nested-loop joins, aggregations, sorts, exchanges), support for the Parquet, ORC/DWRF and Meta’s Nimble file formats, storage adapters for S3, HDFS, GCS and Azure (ABFS), the Presto and Spark wire formats, and memory management with spilling and caching.

Velox Vectors follow the Apache Arrow layout but add 16-byte string views, out-of-order writes for conditionals, and run-length and constant encodings, so filters and joins can often work on encoded data without decoding it. Velox also adapts as it runs: it tracks how selective each filter is and reorders them, and learns which columns to prefetch. Meta started it in late 2020, open-sourced it in 2022, and develops it with IBM, Intel, Voltron Data, Microsoft, ByteDance and others. It is not part of a foundation: the code lives in Meta’s facebookincubator GitHub organization under a project leadership council of individuals. There are no numbered releases; the repository is tagged by date, most recently v2026.10.02.00.

Why teams pick it

  • Large measured gains from replacing JVM execution: in Meta’s VLDB 2022 paper, Prestissimo ran TPC-H Q1 8.4x faster than Presto Java, replayed production queries about 6–7x faster on average, and carried the same workload on 20 servers instead of 60
  • One engine, consistent semantics: Presto and Spark function packages give the same results across systems, where Meta once counted at least 12 different substr() implementations
  • Built to be extended: custom types, scalar, aggregate and window functions, operators, file formats, storage adapters and network serializers
  • Works on encoded data (dictionary, run-length, constant) and delays decoding, which makes filters, joins and unnests over complex types cheaper

Watch out for

  • Not for end users: without Presto, Gluten or your own planner on top there is no way to submit SQL, and plans must arrive fully optimized
  • No versioned releases, only date-stamped tags, and building it means a C++ toolchain (GCC 11 or Clang 15 or newer) and a long list of dependencies; the pyvelox Python package was last published in April 2025
  • Its vectors differ from Arrow in places, so handing data to Arrow tools is zero-copy only when the layouts match; otherwise Velox rearranges the data
  • Meta leads it outside any foundation, so its direction depends on Meta and the companies working with it

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.

How it fits with other technologies

Build a plan by hand and run it on Velox with pyvelox (TPC-H rows generated in memory) · python

# pip install pyvelox==0.2.0   (Python 3.12, macOS arm64)
from pyvelox.plan_builder import PlanBuilder
from pyvelox.runner import LocalRunner, register_tpch

register_tpch("tpch")  # a connector that generates TPC-H rows in memory

# Velox takes an already-optimized plan: there is no SQL parser or optimizer
plan = (
    PlanBuilder()
    .tpch_gen(table_name="lineitem", connector_id="tpch", scale_factor=0.1,
              columns=["l_returnflag", "l_quantity", "l_extendedprice", "l_shipdate"])
    .filter("l_shipdate <= DATE '1998-09-02'")
    .aggregate(grouping_keys=["l_returnflag"],
               aggregations=["sum(l_quantity) AS qty", "avg(l_extendedprice) AS avg_price",
                             "count(1) AS line_items"])
    .order_by(["l_returnflag"])
)

runner = LocalRunner(plan.get_plan_node())
for batch in runner.execute():
    for i in range(batch.size()):
        print([batch.child_at(c)[i] for c in range(4)])
# ['A', '3774200', '38294.602618376986', '147790']
# ['N', '7554554', '38288.665318783744', '295765']
# ['R', '3785523', '38276.82187139619', '148301']

Engines that run on Velox#

Prestissimo (Presto C++)

Presto workers

A C++ worker that speaks Presto’s HTTP protocol, takes plan fragments from the usual Java coordinator, translates them into Velox plans and runs them. No JVM or garbage collector on the workers. Meta started it in 2020.

Apache Gluten

Spark plugin

Created by Intel. Passes Spark plans to a native engine (Velox or ClickHouse) through a JNI interface built on Arrow and Substrait. Meta reported about 3x efficiency gains on benchmarks. A top-level Apache project since March 2026.

Meta’s internal systems

inside Meta

By 2024 Velox was in different stages of integration with more than 10 data systems at Meta, including stream processing, data ingestion and ML feature engineering.

See it in a real stack

No architecture uses Velox yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2020–2026

By 2020 Meta ran dozens of specialized data engines for batch ETL, interactive SQL, streaming, ingestion and machine learning. They shared almost no code, were written in different languages and behaved differently: an informal survey found at least 12 implementations of substr(), with different indexing, null handling and errors. Pedro Pedreira and colleagues noticed that the engines differed mostly in their front ends, optimizers and runtimes, while their execution engines were much alike. In late 2020 Meta started writing one execution layer as a reusable, extensible C++ library, so each optimization would be built once and every engine would get it.

The first full integration was Prestissimo, which replaced Presto’s Java workers with C++ ones; in the VLDB 2022 paper it handled the same production workload with a third of the servers. Meta open-sourced Velox in 2022 and announced it publicly in March 2023, with Ahana, Intel and Voltron Data as key collaborators. Intel built Gluten to run Spark on Velox. Velox’s own vector layout had diverged from Apache Arrow, which hurt interoperability, so Meta and Voltron Data worked with the Arrow community for a year until Arrow added StringView, ListView and run-end encoding, all inspired by Velox Vectors.

Why it’s built this way

Only the execution engine, with no parser or optimizer
The front end and optimizer are what make each engine specialized; the execution layer was the duplicated part. Taking an optimized plan as input lets Presto, Spark, streaming and ML systems all share it.
Extend Arrow instead of adopting it unchanged
Inline string prefixes, out-of-order writes and extra encodings sped up filters, conditionals and joins. The cost was conversion at the boundary, which Meta later reduced by getting similar layouts added to Arrow itself.
Function packages per SQL dialect
Engines need their own semantics (0- or 1-based indexes, null handling, errors), so Velox ships Presto and Spark function sets instead of one generic set.
Vectorization and adaptivity before code generation
Velox leans on tight batch-at-a-time loops and on learning while it runs (reordering filters, prefetching columns). The 2022 paper still called code generation experimental, worth it only where the gains outweigh compile time.

Timeline

  1. 2020

    Meta starts Velox and the Prestissimo project to replace Presto’s Java workers

  2. 2022

    Open-sourced; VLDB paper reports Prestissimo running the same workload on a third of the servers

  3. 2023

    Meta formally announces Velox in March, with Ahana, Intel and Voltron Data as collaborators

  4. 2024

    Meta reports that Arrow has added StringView, ListView and run-end encoding, inspired by Velox Vectors

  5. 2026

    Apache Gluten, which runs Spark on Velox, becomes a top-level Apache project in March

Sources8 · facts checked 11 October 2026

Improve this page on GitHub