Sail

A Spark Connect server written in Rust on Apache DataFusion: your PySpark code connects to it as if it were Spark, and it runs the queries without a JVM, on one machine or on Kubernetes.

What it is: Query engines
Type: Open source
Made by: LakeSail
License: Apache 2.0
Facts checked: 23 October 2026

What it is

Since Spark 3.4, a PySpark program can be a thin client that sends its DataFrame and SQL plans over gRPC to a Spark Connect server, which runs them and returns Arrow batches. Sail is a different server behind the same protocol. It parses the Spark SQL and DataFrame plans with its own Rust parser and analyzer, plans and runs them on Apache DataFusion over Apache Arrow batches, and streams the results back. You change the connection string to sc://host:50051 and keep the code. Sail works with Spark 3.5 and Spark 4 clients, including 4.2, and with the small pyspark-client package, which has no JARs.

In local mode Sail is one process that runs each query on all CPU cores. In cluster mode a driver splits the plan into stages and sends tasks to stateless workers, which Sail starts as pods on Kubernetes and stops when idle. Shuffle data moves between workers over Arrow Flight, or, since 0.7, can be written to object storage first so a failed task doesn’t rerun earlier stages. It reads and writes Delta Lake and Iceberg tables natively, uses Iceberg REST, AWS Glue, Unity Catalog, Hive Metastore or OneLake as catalogs, and runs Python, Pandas and Arrow UDFs. LakeSail, the company behind it, releases it under Apache 2.0; the current version is 0.7.2 (September 2026).

Why teams pick it

No rewrite: existing PySpark and Spark SQL code runs against it by changing the remote address, and a script can start the server itself with pip install pysail
No JVM: one Rust binary with no garbage collector or JVM tuning; LakeSail’s derived TPC-H run at scale factor 100 took 53 s and 26 GB peak memory against Spark’s 535 s and 72 GB (its own benchmark)
Same engine from laptop to cluster: local mode for development and tests, cluster mode with workers on Kubernetes for larger jobs
Native Delta Lake and Iceberg reads and writes, including MERGE on Iceberg and UPDATE on Delta in recent releases

Watch out for

Only what Spark Connect carries: the RDD API and SparkContext aren’t supported, pandas-on-Spark is still planned, and Structured Streaming is only partly supported
Many Spark settings have no effect, Hadoop storage options such as s3a settings are ignored, and error messages differ from Spark’s
Still 0.x: the docs warn that configuration isn’t stable yet, and on Kubernetes each Spark session is tied to one server pod, so the server runs as a single replica
Compatibility is checked per feature: the bundled scanner reports which PySpark functions your code uses that Sail implements, but not whether results match Spark’s

Good for

Scans and aggregations over billions of rowsRuns Spark SQL and DataFrame plans on DataFusion over Arrow batches, on all cores of one machine or across worker pods on Kubernetes.

Specs

Compare with alternatives →

Data model: Spark DataFrames and SQL tables, including Delta Lake and Iceberg
Query language: Spark SQL and the PySpark DataFrame API
Client protocol: Spark Connect (gRPC), returning Arrow batches
How it scales: One process on all cores, or a driver with stateless worker pods on Kubernetes
Written in: Rust
Ways to run it: Self-hosted, as a local process or on Kubernetes

Alternatives

What all query engines have in common →Compare side by side →

Apache SparkOpen sourceAlso good for scans and aggregations over billions of rowsThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.+1 Apache Gluten

Apache GlutenOpen sourceAlso good for scans and aggregations over billions of rowsA Spark plugin that hands Spark’s physical plans to a native C++ engine, Velox or ClickHouse, through Substrait, so SQL and DataFrame jobs run vectorized with no code changes, and falls back to Spark for anything the engine can’t run.

Apache DataFusion CometOpen sourceAlso good for scans and aggregations over billions of rowsA Spark plugin that keeps Spark’s optimizer and scheduler but runs the physical plan on Apache DataFusion in Rust, over Arrow batches, with no changes to your Spark code, and falls back to Spark for anything it can’t run the same way.

Apache DataFusionOpen sourceAlso good for scans and aggregations over billions of rowsA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion Comet

TrinoOpen sourceAlso good for scans and aggregations over billions of rowsA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.

Presto (PrestoDB)Open sourceAlso good for scans and aggregations over billions of rowsFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.

StarRocksOpen sourceAlso good for scans and aggregations over billions of rowsAn open-source MPP analytical database with a vectorized engine that serves fast joins and real-time updates on its own tables, and also queries Iceberg, Hudi, Delta Lake and Hive tables in place.

Apache DorisOpen sourceAlso good for scans and aggregations over billions of rowsAn open-source MPP analytical database, born at Baidu as Palo, that serves sub-second queries on fresh, updatable data through the MySQL protocol and also queries Iceberg, Hudi, Paimon and Hive tables in place.

DremioCommercialA SQL lakehouse engine built on Apache Arrow, with its own Iceberg catalog and “reflections” that speed up queries without anyone changing them.

DatabricksCommercialA managed lakehouse platform built on Apache Spark and Delta Lake, with Unity Catalog for governance, on AWS, Azure or Google Cloud.

How it fits with other technologies

What Sail uses

Start a Sail server inside a script and use it from ordinary PySpark · python

# pip install "pysail==0.7.2" "pyspark-client==4.2.0"
# (or run `sail spark server` and connect to sc://localhost:50051)
from pysail.spark import SparkConnectServer
from pyspark.sql import SparkSession

server = SparkConnectServer()
server.start()
_, port = server.listening_address

spark = SparkSession.builder.remote(f"sc://localhost:{port}").getOrCreate()
spark.sql("SELECT 1 + 1").show()

See it in a real stack

No architecture uses Sail yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2024–2026

Spark is the default engine for large data jobs, but it runs on the JVM, so clusters need memory and tuning. Projects such as Gluten and DataFusion Comet keep Spark and replace only its execution with native code. Spark Connect, added in Spark 3.4, opened another route: clients send unresolved query plans as protocol buffers over gRPC, so a different server can answer them.

Shehab Amin and Heran Lin of LakeSail released Sail 0.1 as the pysail Python package in August 2024: “a drop-in replacement for Spark SQL and the DataFrame API” in a single process, written in Rust on Arrow and DataFusion. Its stated mission is to unify batch, stream and compute-intensive AI workloads, starting with Spark because so many people use it. A preview of distributed execution on Kubernetes followed in November 2024.

Later releases filled in the Spark surface and the lakehouse: an MCP server for LLM agents in March 2025, Spark 4.0 in 0.3 (July 2025), native Iceberg built on iceberg-rust in 0.4 (October 2025), more resilient distributed execution in 0.5 (February 2026), and in 0.7 (August 2026) Spark 4.2, blocking shuffle through object storage and checkpoints.

Why it’s built this way

Be a Spark Connect server: Clients already speak the protocol, so users keep their PySpark code and tools and only point them at a different address. The cost is that anything outside Spark Connect, such as the RDD API, can’t be supported.
Rust on DataFusion and Arrow, no JVM: The 0.2 announcement cites memory safety, built-in async/await with Tokio for the control plane, and serialization-free data transfer with Arrow, which the authors tie to higher throughput and lower memory use.
An actor model for the driver and workers: LakeSail wrote a small actor framework so the driver and workers keep state without locks, following “Don’t communicate by sharing memory; share memory by communicating.”
Stateless workers, job state in storage: Sail 0.7 treats separate storage and compute as its home: shuffle data and checkpoints go to object storage, so a failed or preempted worker doesn’t force upstream stages to rerun, and small clusters can run big jobs one stage at a time.

Timeline

2024
Sail 0.1 ships as the pysail package in August
2024
0.2 previews distributed execution on Kubernetes in November
2025
An MCP server lets LLM agents run Spark SQL on Sail (March)
2025
0.3 adds Spark 4.0 (July); 0.4 adds native Iceberg (October)
2026
0.5 makes distributed execution resilient and observable (February)
2026
0.7 adds Spark 4.2, blocking shuffle through object storage and checkpoints (August)

Sources9 · facts checked 23 October 2026

Improve this page on GitHub