Cloud Parts

Apache PigDormant

A step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.

What it is
Query engines
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Status
Dormant. The project described itself as dormant in its April 2026 board report: maintenance only.
Facts checked
6 October 2026
Website Source code

What it is

You write a Pig Latin script as a sequence of named steps: LOAD a file, FILTER, GROUP, JOIN, FOREACH ... GENERATE, then STORE the result. Pig runs on the client: it parses the script into a logical plan, optimizes it, and compiles it into a chain of MapReduce jobs (the default), a Tez DAG (-x tez) or Spark jobs (-x spark) that it submits to the cluster. Data stays in plain files in HDFS; load and store functions parse and write them, and no schema is required up front. Pig’s data model is nested (tuples, bags and maps), and user-defined functions in Java or Python plug into any step. The interactive Grunt shell and -x local mode run scripts on one machine for testing.

Pig is free under Apache 2.0 and is not retired, but it is barely moving. After 0.17.0 in June 2017 there was no release for eight years, until 0.18.0 in September 2025 added support for Hadoop 3, Tez 0.10, Spark 3, Hive 3, HBase 2 and Python 3. In its April 2026 board report the project described its status as “Dormant”, in maintenance mode with bug fixes and dependency updates, and no new committer since 2018. A 0.19.0 release was targeted for May 2026 but has not appeared. Amazon EMR still offers Pig, at version 0.17.0.

Why teams pick it

  • Readable dataflow scripts: each line is one transformation, which suits multi-step ETL better than one large SQL statement
  • Works on raw files with no schema or loading step, and handles nested data natively
  • UDFs in Java or Python plug into any step, and Pig can call Hive UDFs
  • Existing scripts can move from MapReduce to Tez or Spark by changing the -x flag

Watch out for

  • Dormant: one release in eight years, no new committers since 2018, and maintenance limited to bug fixes and dependency updates
  • Docs lag the code: the 0.18.0 getting-started page still lists Hadoop 2.X and Java 1.7 as requirements
  • Tied to Hadoop: cluster modes need a Hadoop cluster and HDFS, and the local Tez and Spark modes are still marked experimental
  • Pig Latin is its own language, not SQL: analysts and BI tools can’t query data through Pig
  • Managed platforms lag too: Amazon EMR 7.14 ships Pig 0.17.0 from 2017

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

Top pages by visits with a Pig Latin script, run locally then on Tez · bash

cat > top_pages.pig <<'EOF'
visits = LOAD 'visits.tsv' USING PigStorage('\t')
         AS (user:chararray, url:chararray, ms:int);
slow    = FILTER visits BY ms > 100;
by_url  = GROUP slow BY url;
counts  = FOREACH by_url GENERATE group AS url, COUNT(slow) AS n,
                                  AVG(slow.ms) AS avg_ms;
ranked  = ORDER counts BY n DESC;
top10   = LIMIT ranked 10;
STORE top10 INTO 'top_pages' USING PigStorage(',');
EOF

# test on one machine against local files
pig -x local top_pages.pig && cat top_pages/part-*

# run the same script on the cluster as a Tez DAG instead of MapReduce jobs
pig -x tez top_pages.pig

Execution modes#

The same script runs on different engines. The 0.18.0 docs list six modes:

ModeFlagNeedsStatus in the docs
Local-x localOne machine and its local file systemSupported
MapReduce-x mapreduce (default)A Hadoop cluster and HDFSSupported
Tez-x tezA Hadoop cluster and HDFSSupported; added in 0.14.0 (2014)
Tez local-x tez_localOne machineExperimental
Spark-x sparkA Spark, YARN or Mesos cluster and HDFSSupported; added in 0.17.0 (2017)
Spark local-x spark_localOne machineExperimental

See it in a real stack

No architecture uses Apache Pig yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2007–2025

Internet companies such as Yahoo were analyzing terabytes of web crawls, search logs and click streams every day. Parallel databases such as Teradata could do it but were prohibitively expensive at web scale, and many of the people doing the analysis were procedural programmers who found SQL unnatural. Hadoop’s MapReduce was cheap and popular, but too low-level and rigid: a one-input, two-stage model that led to lots of custom code that was hard to maintain and reuse.

Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi Kumar and Andrew Tomkins at Yahoo Research designed Pig Latin to sit between the two: a sequence of high-level steps, each a single relational-style transformation, compiled into MapReduce jobs. The prototype won internal users, Yahoo built a production engineering team around it, and in September 2007 Pig entered the Apache Incubator. By June 2009 roughly 60% of Yahoo’s ad-hoc Hadoop jobs and 40% of its production Hadoop jobs were submitted through Pig.

Pig became a top-level Apache project in September 2010, at the same board meeting as Hive. It added Tez in 2014 and Spark in 2017 as faster engines, then went eight years without a release until 0.18.0 in 2025.

Why it’s built this way

A dataflow of steps instead of one declarative query
Yahoo’s programmers preferred writing step-by-step scripts. Using relational primitives such as FILTER and GROUP for each step still let the system reorder and optimize them.
Query raw files with no import step
Ad-hoc analysis started from dumps of logs. Pig reads files in place through a user-supplied parsing function, so there is no loading phase, and other tools keep using the same files.
A fully nested data model
Programmers think in nested structures and much of the data on disk was already nested. Tuples, bags and maps avoided flattening everything into normalized tables.
UDFs as first-class citizens
Analysis at Yahoo needed custom logic such as spam detection or query expansion. Every Pig Latin operation can call user-defined functions, which use the same nested data model.

Timeline

  1. 2007

    Yahoo Research’s Pig enters the Apache Incubator in September

  2. 2008

    SIGMOD paper on Pig Latin; Pig 0.1.0; graduates as a Hadoop subproject

  3. 2009

    About 60% of Yahoo’s ad-hoc Hadoop jobs come through Pig

  4. 2010

    Becomes a top-level Apache project in September

  5. 2014

    Pig 0.14.0 adds Tez as an execution engine

  6. 2017

    Pig 0.17.0 adds Spark as an engine; then eight years without a release

  7. 2025

    Pig 0.18.0 adds Hadoop 3, Spark 3 and Python 3 support

Sources8 · facts checked 6 October 2026

Improve this page on GitHub