Cloud Parts

Apache Spark

The general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.

What it is
Query engines,Stream processing
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
5 October 2026
Website Source code

What it is

A Spark application has a driver, which turns your DataFrame code or SQL into a plan of stages, and executors, which run each stage as many parallel tasks, one per partition of the data. Between stages, data is shuffled across the network, and it spills to disk when it doesn’t fit in memory, so a job can process far more data than the cluster has RAM. If an executor dies, Spark recomputes the partitions it lost. The same engine runs batch jobs, Structured Streaming and MLlib.

Spark runs on Kubernetes, YARN or its own standalone cluster manager, and as managed services such as Databricks, Amazon EMR, AWS Glue and Google Cloud’s Managed Service for Apache Spark (formerly Dataproc). Table formats plug in as libraries: Iceberg, Delta Lake and Hudi each ship a Spark runtime. Spark 4.0 (May 2025) made ANSI SQL mode the default, added a VARIANT type for semi-structured data and moved to Java 17 and Scala 2.13. Spark Connect lets thin clients, including a 1.5 MB Python client, use a remote cluster. The current release is 4.2.0, from July 2026.

Why teams pick it

  • One engine for ETL, SQL, streaming and machine learning
  • Scales from a laptop to clusters of thousands of nodes, and recomputes lost partitions when an executor fails
  • Iceberg, Delta Lake and Hudi each ship a Spark integration that reads and writes their tables
  • Managed services on every major cloud, such as Amazon EMR, Google Cloud’s Managed Service for Apache Spark (formerly Dataproc) and Databricks

Watch out for

  • Built for clusters: each application starts its own driver and executor processes before any work begins
  • Tuning shuffles, partition counts and memory is a skill of its own; skewed keys cause stragglers and out-of-memory errors
  • Spark 4 dropped Java 8 and 11 and Scala 2.12, so upgrading from 3.x means dependency work

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

What uses Apache Spark

Stream a Kafka topic into an Iceberg table with Structured Streaming · python

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("orders-stream").getOrCreate()

orders = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "kafka:9092")
    .option("subscribe", "orders")
    .load()
    .select(F.from_json(F.col("value").cast("string"),
            "order_id BIGINT, amount DECIMAL(10,2), ts TIMESTAMP").alias("o"))
    .select("o.*"))

(orders.writeStream.format("iceberg")
    .option("checkpointLocation", "s3://acme-lake/checkpoints/orders")
    .trigger(processingTime="1 minute")
    .toTable("lake.sales.orders"))

See it in a real stack

Open lakehouse on Iceberg →
Live model
What you’re watchingServices publish order events to Kafka. Spark reads them in micro-batches and commits them to an Iceberg table whose files live on S3. Dashboard queries run on Trino, which reads the same table.

Benchmarks

Benchmark · ClickHouse Inc.ClickBenchHow fast analytical databases run 43 SQL queries over one 100-million-row table of web analytics events, on the same cloud machines.Includes Apache Spark · 7 caveats to read first
How to read a benchmark →
Where it came from2009–2026

By 2009, Hadoop MapReduce was the standard way to process big data on a cluster, but it had one habit that hurt: every job read its input from disk and wrote its output back. Machine learning algorithms that pass over the same data dozens of times reloaded it on every pass, and an analyst running ad-hoc queries through Hive or Pig waited tens of seconds for each one, because each query was a separate MapReduce job.

Spark was created that year in UC Berkeley’s AMPLab by Matei Zaharia, working with Ion Stoica, Scott Shenker and others. The idea was to keep a working set of data in memory across many parallel operations while keeping MapReduce’s scalability and fault tolerance. The first paper, at HotCloud 2010, reported Spark running iterative machine learning 10x faster than Hadoop and scanning a 39 GB dataset interactively with sub-second latency. The code was released under a BSD licence in 2010, the 2012 NSDI paper on Resilient Distributed Datasets won Best Paper, and in 2013 the creators founded Databricks and moved Spark to Apache.

Libraries for SQL, streaming, machine learning and graphs grew on top of the same core.

Why it’s built this way

Keep data in memory between steps
Iterative algorithms and repeated queries spent most of their time rereading the same input from disk. Caching a dataset in RAM across a cluster removed that cost for multi-pass work.
Recover lost data from lineage, not replicas
Replicating every in-memory dataset would double memory use and network traffic. An RDD instead remembers the coarse transformations (map, join, groupBy) that built it, so Spark can recompute just a lost partition.
Written in Scala, usable from an interactive shell
Scala ran on the JVM next to Hadoop but offered functional operations and closures that could be shipped to workers. A modified Scala interpreter let users explore large datasets on a cluster interactively.
One engine for batch, SQL, streaming and ML
Instead of separate systems for each workload, Shark (later Spark SQL), Spark Streaming, MLlib and GraphX were built on the same execution core, so data didn’t have to move between systems.

Timeline

  1. 2009

    Spark created at UC Berkeley’s AMPLab

  2. 2010

    HotCloud paper in June; BSD licence added to the code in December

  3. 2013

    Creators found Databricks; Spark enters the Apache Incubator in June

  4. 2014

    Top-level Apache project in February; 1.0 in May; sets a sorting record

  5. 2016

    Spark 2.0 unifies DataFrames and Datasets and previews Structured Streaming

  6. 2025

    Spark 4.0 makes ANSI SQL the default and adds VARIANT

  7. 2026

    Spark 4.2 adds geospatial types and change data capture

Sources17 · facts checked 5 October 2026

Improve this page on GitHub