Cloud Parts

Apache DataFusion Comet

A Spark plugin that keeps Spark’s optimizer and scheduler but runs the physical plan on Apache DataFusion in Rust, over Arrow batches, with no changes to your Spark code, and falls back to Spark for anything it can’t run the same way.

What it is
Query engines
Part of
Built on Apache DataFusion
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
11 October 2026
Website Source code

What it is

Comet plugs into Spark through spark.plugins and its own shuffle manager. Spark still parses, optimizes and schedules every query. Comet then walks the physical plan and swaps each operator it supports for a Comet* version: Parquet scans (including the files behind Iceberg tables), filters, projections, hash aggregates, all four Spark join types, sorts, window functions, explode, and a native Arrow IPC shuffle. Most run as native Rust code built on DataFusion inside the executor process; some run in the JVM directly on Arrow batches. Data stays in Arrow columnar format from scan to shuffle to result, instead of Spark’s row format. You can see what happened in explain(): Comet* nodes are accelerated, and nodes without the prefix are plain Spark.

Matching Spark’s results is the hard part. Every supported expression runs in a Spark-compatible way by default; native versions that differ for some inputs stay off unless you opt in per expression. When an operator, expression or data type isn’t supported, that part of the plan falls back to Spark, with row and column conversions at the boundaries. Comet runs Spark’s own test suite, more than 24,000 tests, in CI. It needs JDK 17 or newer and off-heap memory, supports Spark 3.4 (deprecated), 3.5, 4.0 and 4.1, with 4.2 experimental, and its published jars contain native code for Linux on amd64 and arm64 only. Apple built Comet and donated it in 2024. It became 1.0 in August 2026 and now follows semantic versioning; the current release is 1.1.0 (October 2026), built on DataFusion 55.1 and Arrow 59.2.

Why teams pick it

  • No code changes: add one jar and a few settings to an existing Spark job, and SQL, DataFrame and PySpark code run as before
  • Measured gains on commodity hardware: the project reports about 1.8x faster TPC-DS at 1 TB than Spark 4.2 (about 45% lower cost), with no GPUs or special hardware
  • Conservative about correctness: incompatible native paths are off by default, unsupported pieces fall back to Spark, and Spark’s own test suite runs against it
  • Covers the expensive parts: Parquet and Iceberg scans, joins, aggregates, windows and a native shuffle, with more than 400 Spark expressions

Watch out for

  • Speedups depend on coverage: each fallback adds row-to-column conversions, and frequent transitions can erase the gain, so you need to read plans or turn on spark.comet.explain.fallback.enabled
  • Published jars are for Linux only (amd64, arm64); macOS needs a source build, and Comet may not fully work on cloud vendors’ proprietary Spark forks
  • Needs off-heap memory sized for native execution (1.1.0 disables itself without it), plus executor memory overhead tuning
  • Some edge cases still differ from Spark, documented per feature: for example, dates before 15 October 1582 in Parquet files written with Spark’s legacy calendar are read without rebasing and come out wrong, silently

How it fits with other technologies

Turn Comet on for an ordinary PySpark job and see Comet operators in the plan · python

# spark-submit --jars comet-spark-spark4.1_2.13-1.1.0.jar \
#   --conf spark.driver.extraClassPath=comet-spark-spark4.1_2.13-1.1.0.jar \
#   --conf spark.executor.extraClassPath=comet-spark-spark4.1_2.13-1.1.0.jar \
#   comet_demo.py                      (Spark 4.1.3, Java 17, Linux arm64)
from pyspark.sql import SparkSession

# The same Spark code as always; only the session config adds Comet
spark = (SparkSession.builder
         .config("spark.plugins", "org.apache.spark.CometPlugin")
         .config("spark.shuffle.manager",
                 "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
         .config("spark.memory.offHeap.enabled", "true")
         .config("spark.memory.offHeap.size", "2g")
         .getOrCreate())

spark.range(1_000_000).selectExpr("id % 100 AS store", "id * 0.5 AS amount") \
     .write.mode("overwrite").parquet("/tmp/sales")

df = spark.sql("""
    SELECT store, sum(amount) AS revenue
    FROM parquet.`/tmp/sales` WHERE amount > 100
    GROUP BY store ORDER BY revenue DESC LIMIT 3
""")
df.explain()
# +- CometTakeOrderedAndProjectExec(limit=3, ...)
#    +- CometHashAggregate [...], [Final], [store#4L], [sum(amount#5)]
#       +- CometExchange hashpartitioning(store#4L, 200), ..., CometNativeShuffle
#          +- CometHashAggregate [...], [Partial], [store#4L], [partial_sum(amount#5)]
#             +- CometFilter [store#4L, amount#5], (isnotnull(amount#5) AND (amount#5 > 100.0))
#                +- CometNativeScan parquet [store#4L,amount#5] ...
df.show()
# |store|     revenue|
# |   99|2500244851.0|
# |   98|2500239852.0|
# |   97|2500234853.0|

Spark versions in Comet 1.1#

From the Comet 1.1.0 installation guide, checked 2026-10-11. Comet needs JDK 17 or newer.

SparkJavaScalaStatus
3.4.3172.12, 2.13Deprecated since 1.0.0; Spark’s SQL tests run only on demand
3.5.9172.12, 2.13Supported; Comet and Spark SQL tests nightly
4.0.417, 212.13Supported; tests nightly
4.1.317, 212.13Supported; tests must pass before every merge
4.2.0172.13Experimental, for development and testing only

See it in a real stack

No architecture uses Apache DataFusion Comet yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2024–2026

Spark runs on the JVM and processes rows in its built-in engine. Several projects set out to keep Spark’s planner and APIs while running the heavy work natively in columnar form: Databricks’ proprietary Photon, and open-source Gluten, Spark RAPIDS and Blaze. Comet was one of them, first implemented at Apple by engineers who were also significant contributors to Arrow and DataFusion. In March 2024 the Arrow PMC announced that Apple had donated it, under the name Arrow DataFusion Comet, as a plugin that hands Spark’s physical plan to DataFusion while queries still go through Spark’s optimizer and planner.

The first Apache release, 0.1.0 in July 2024, came out of five months of work by 41 contributors and supported 13 operators and 106 expressions on Spark 3.3 to 3.5. Early releases gave modest speedups. Over the next two years the project shipped 20 releases, drew more than 120 contributors and grew to over 400 Spark expressions, a fully native shuffle and Iceberg support. Release 1.0.0 in August 2026, with full support for Spark’s ANSI mode, started semantic versioning; 1.1.0 in October 2026 added experimental native Iceberg writes, reworked memory management and dropped JDK 11.

Why it’s built this way

Keep Spark’s planner and replace only execution
Users keep the same Spark ecosystem, tools and APIs; only the work inside each executor moves to DataFusion, so adopting it needs no code changes.
Fall back rather than return different results
From the first release the goal was 100% compatibility with Spark: anything not fully compatible falls back to Spark unless the user explicitly enables it.
Arrow from scan to shuffle
Keeping operators, expressions, shuffle and broadcast in Arrow columnar format avoids the per-row overhead of Spark’s row-based engine and repeated conversions.
Test against Spark’s own suite
Running Spark’s 24,000-plus unit tests with Comet enabled, alongside fuzzing and Comet’s own SQL tests, is how the project checks that results match Spark.

Timeline

  1. 2024

    Apple donates Comet to the Apache Arrow project; announced 6 March

  2. 2024

    0.1.0 in July: 13 operators, 106 expressions, Spark 3.3 to 3.5

  3. 2026

    0.17.0 in June runs expressions it lacks through Spark’s generated code instead of falling back

  4. 2026

    1.0.0 in August: semantic versioning, ANSI mode, over 400 expressions

  5. 2026

    1.1.0 in October: experimental native Iceberg writes, JDK 17 or newer required

Sources9 · facts checked 11 October 2026

Improve this page on GitHub