Apache Gluten

A Spark plugin that hands Spark’s physical plans to a native C++ engine, Velox or ClickHouse, through Substrait, so SQL and DataFrame jobs run vectorized with no code changes, and falls back to Spark for anything the engine can’t run.

Apache · Top-level project

What it is: Query engines
Part of: Built on Apache Spark
Type: Open source
Made by: Apache Software Foundation
Foundation: The Apache Software Foundation, top-level project, joined January 2024
License: Apache 2.0
Facts checked: 23 October 2026

What it is

Gluten loads into Spark through spark.plugins and its own columnar shuffle manager. Spark still parses, optimizes and schedules the query. Gluten then converts the physical plan into a Substrait plan, passes it over JNI to a native library in each executor, and gets results back as columnar batches in Arrow format through Spark’s columnar API. Two backends do the work: Meta’s Velox and ClickHouse’s engine, and the docs now also cover a newer backend called Bolt. Operators the backend can’t run fall back to ordinary Spark, with row-to-column conversions at the edges. Gluten also manages native memory together with Spark’s, reuses Spark’s shuffle service for its columnar shuffle, and shows native metrics in the Spark UI.

Intel and Kyligence started Gluten in 2022; the name is Latin for glue. It entered the Apache Incubator in December 2023 and graduated to a top-level project in February 2026. Version 1.7.0 (August 2026) publishes Velox bundles for Spark 3.3 to 4.1, built on CentOS 7 with static linking for x86-64 Linux, and adds native reading of Delta Lake deletion vectors and native Delta writes on Spark 4.0. The project reports 3.34x faster TPC-H and 3.02x faster TPC-DS than vanilla Spark at 3 TB with Velox, on its own Intel Xeon hardware.

Why teams pick it

No code changes: add the bundle jar, the plugin, off-heap memory and the columnar shuffle manager to an existing Spark job
Large measured gains on scan-, join- and aggregate-heavy SQL: about 3x on TPC-H and TPC-DS at 3 TB in the project’s own runs, with single queries up to 23x faster
Choice of native engine: Velox, shared with Presto’s C++ workers, or ClickHouse’s engine, behind the same plugin
A broad community: the first committers came from Intel, Kyligence, Alibaba, Meituan, BIGO, Baidu, NetEase and Microsoft, and the project lists more than 15 production adopters

Watch out for

Gains depend on coverage: with ANSI mode on, the whole plan falls back to Spark; non-Parquet scans (ORC only partly) and bucketed writes fall back too, and each fallback adds conversions
Some results can differ from Spark’s: the Velox backend ignores Parquet datetime rebasing, doesn’t support NaN in some cases, accepts only double-quoted JSON strings and has no regex lookaround
Binary bundles are built for one Linux and CPU baseline, and the project says performance and compatibility aren’t guaranteed elsewhere, so many users build from source
Needs off-heap memory sized for native execution; with many shuffle partitions it can still run out of memory despite spilling

Good for

Scans and aggregations over billions of rowsRuns Spark’s scans, joins and aggregates on Velox or ClickHouse’s vectorized C++ engine; about 3x faster TPC-H and TPC-DS in its own runs.

Specs

Compare with alternatives →

Query language: Spark SQL and DataFrame code, unchanged
How it scales: Runs inside Spark executors; Spark plans and schedules the cluster
Storage format: Parquet natively, ORC partly; native Delta Lake deletion vectors and writes
Written in: Scala and C++
Ways to run it: A bundle jar for Spark 3.3 to 4.1, built for x86-64 Linux, or built from source

More of the Apache Spark family

What all query engines have in common →Compare side by side →

Apache DataFusion CometOpen sourceBuilt on itA Spark plugin that keeps Spark’s optimizer and scheduler but runs the physical plan on Apache DataFusion in Rust, over Arrow batches, with no changes to your Spark code, and falls back to Spark for anything it can’t run the same way.

SailOpen sourceAlso good for scans and aggregations over billions of rowsA Spark Connect server written in Rust on Apache DataFusion: your PySpark code connects to it as if it were Spark, and it runs the queries without a JVM, on one machine or on Kubernetes.

Apache SparkOpen sourceAlso good for scans and aggregations over billions of rowsThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.+1 Apache Gluten

VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

What Apache Gluten uses

What uses Apache Gluten

LakeSoulcan speed up Spark reads withApache GlutenTable formats

Turn Gluten on for a Spark shell with the Velox bundle (from the project’s home page) · bash

spark-shell \
  --master yarn --deploy-mode client \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=20g \
  --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
  --jars gluten-velox-bundle-spark3.5_2.12-centos_7_x86_64-1.6.0.jar

See it in a real stack

No architecture uses Apache Gluten yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2022–2026

Spark runs on the JVM and processes data in its own row-based engine, while native C++ engines such as Velox and ClickHouse run vectorized code much closer to the hardware. Moving to another engine would mean rewriting Spark jobs, and Spark’s planning, scheduling and scalability were worth keeping.

Intel and Kyligence started Gluten in 2022 to “glue” Spark to native libraries: reuse Spark’s control flow and JVM code, and run only the compute-heavy data processing natively. They released it as an open-source Spark plugin, with Velox and ClickHouse backends, and shipped 1.0 in July 2023. The incubator proposal listed initial committers from Intel, Kyligence, Alibaba, Meituan, BIGO, Baidu, NetEase and Microsoft.

Gluten entered the Apache Incubator on 15 December 2023, made its first Apache release, 1.2.0, in September 2024, and graduated to a top-level project on 18 February 2026. Release 1.7.0 in August 2026 dropped Spark 3.2, added Spark 4.1 and JDK 21 and 25 builds, and added native Delta Lake deletion-vector reads.

Why it’s built this way

Keep Spark’s planner and scheduler, replace only execution: The proposal’s goal was better performance without changing existing Spark SQL jobs, so users keep the same APIs, cluster managers and tools.
Substrait as the hand-off format: Spark’s physical plan is translated into Substrait, a plan format not tied to one engine, and each backend runs that plan. This is how one plugin drives both Velox and ClickHouse.
Fall back to Spark for anything unsupported: Operators, expressions or formats the backend can’t run go back to vanilla Spark, so a job still completes and gets faster where coverage exists.
Arrow columnar batches through Spark’s columnar API: Native results come back as ColumnarBatch objects in Arrow format, which Spark already knows how to pass between operators, and a columnar shuffle avoids converting to rows between stages.

Timeline

2022
Intel and Kyligence start Gluten as an open-source Spark plugin
2023
Gluten 1.0.0 in July; enters the Apache Incubator on 15 December
2024
First Apache release, 1.2.0, in September
2026
Graduates to a top-level Apache project on 18 February
2026
1.7.0 in August: Spark 4.1, JDK 21 and 25, native Delta deletion vectors

Sources8 · facts checked 23 October 2026

Improve this page on GitHub