Cloud Parts

Apache Hive

The first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.

What it is
Query engines,Data warehouses
Used for
Data lakes
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
6 October 2026
Website Source code

What it is

Hive turns SQL (HiveQL) into distributed jobs over files. A table is a directory in HDFS or an object store, and each partition is a subdirectory such as ds=2026-10-05. The Hive Metastore (HMS) is a Thrift service, port 9083 by default, that keeps each table’s schema, location, partitions, file format and statistics in a relational database such as PostgreSQL or MySQL. Clients connect to HiveServer2 over JDBC or ODBC, for example with beeline; it plans each query with Apache Calcite and runs it as a DAG of Apache Tez tasks in YARN containers. MapReduce, the original engine, has been deprecated since Hive 2.0, and Hive on Spark was removed from the main branch before Hive 4. LLAP, added in Hive 2.0, keeps long-lived daemons that cache column data and run short queries directly.

Tables can be plain files (ORC, Parquet, Avro, text), Hive ACID tables, which support INSERT, UPDATE, DELETE and MERGE on ORC files and need compaction, or Iceberg tables created with STORED BY ICEBERG. Hive is free under Apache 2.0. The current line is 4.2 (4.2.0 in November 2025, 4.2.1 in August 2026), which requires JDK 21; the 1.x, 2.x and 3.x lines were declared end of life in 2024. Since 4.1 the Metastore also ships as a standalone binary and Docker image, with a REST catalog server. Amazon EMR still ships Hive 3.1.3, and Cloudera Data Warehouse sells Hive as one of its two Virtual Warehouse engines, next to Impala.

Why teams pick it

  • The Hive Metastore is the most widely supported table catalog: Trino, Spark, Impala, Flink and Iceberg’s HiveCatalog all use it, and AWS Glue implements its API
  • Batch SQL over petabytes of data on YARN; the project describes it as a distributed, fault-tolerant warehouse
  • Familiar SQL over JDBC and ODBC through HiveServer2, with Kerberos authentication, fine-grained access control and audit logging
  • Hive 4 reads and writes Iceberg tables, including UPDATE, DELETE, MERGE, time travel and compaction

Watch out for

  • Built for batch, not interactive dashboards: queries run as Tez DAGs in YARN containers, and LLAP daemons exist precisely so short queries can skip that
  • A Hadoop stack to run: HiveServer2, the Metastore and its database, YARN and Tez; the embedded Derby metastore loses everything when the container stops
  • Engines have come and gone: MapReduce deprecated since 2.0 (its removal is still an open ticket), Hive on Spark removed before 4.0; on Iceberg tables, DML works only on Tez
  • Hive ACID tables are ORC-only and depend on background compaction; upgrading to 3.0 required a major compaction of every changed partition
  • Old lines are end of life: 3.x since October 2024, yet Amazon EMR 7.14 still ships Hive 3.1.3, and Hive 4.2 needs JDK 21

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

A partitioned ORC table and an Iceberg table in Hive 4 · sql

-- docker run -d -p 10000:10000 --env SERVICE_NAME=hiveserver2 --name hive4 apache/hive:4.2.1
-- docker exec -it hive4 beeline -u 'jdbc:hive2://localhost:10000/'

-- classic Hive table: one HDFS directory per day, files in ORC
CREATE TABLE page_views (user_id BIGINT, url STRING, ms INT)
PARTITIONED BY (ds STRING)
STORED AS ORC;

INSERT INTO page_views PARTITION (ds = '2026-10-05')
VALUES (42, '/pricing', 180), (7, '/docs', 95);

-- the same data as an Iceberg table with a hidden daily partition
CREATE TABLE page_views_ice (user_id BIGINT, url STRING, ms INT, ts TIMESTAMP)
PARTITIONED BY SPEC (day(ts))
STORED BY ICEBERG;

INSERT INTO page_views_ice
SELECT user_id, url, ms, CAST(ds AS TIMESTAMP) FROM page_views;

DELETE FROM page_views_ice WHERE user_id = 42;

Execution engines over time#

Hive has changed how it runs queries three times. The SQL stayed the same; what runs underneath did not.

EngineAddedStatus in Hive 4
MapReduceThe original engine (2008)Deprecated since Hive 2.0; a ticket to remove it has been open since January 2024
TezHive 0.13 (April 2014)The engine to use: the project asks all users to move workloads to Tez, and Iceberg DML needs it
SparkHive 1.1 (March 2015)Removed from the main branch in April 2022, before Hive 4.0
LLAP daemonsHive 2.0 (February 2016)Long-lived daemons that cache columns and run small queries; heavy work still goes to YARN containers

The Metastore’s afterlife#

Many teams that never run a HiveQL query still run a Hive Metastore, because it is the catalog the rest of the ecosystem learned to speak.

Trino and Presto

Hive connector

Trino’s Hive connector uses only Hive’s data and metadata, not HiveQL or its runtime. It needs a Hive Metastore over Thrift (port 9083) or a compatible service such as AWS Glue.

Apache Spark

enableHiveSupport()

Spark bundles a Hive 2.3.10 metastore client and can talk to Metastore versions 2.0 through 4.1, so CREATE TABLE ... USING hive lands in the same catalog Hive uses.

Apache Iceberg

HiveCatalog

Iceberg can use the Metastore as its catalog (HiveCatalog), next to options such as AWS Glue, JDBC and REST catalogs. The integration supports Hive 2.3.10, 3.1.3 or later.

Apache Flink and Impala

shared catalog

Flink’s HiveCatalog stores Flink table definitions in the Metastore; Flink’s docs call it the only persistent catalog Flink provides out of the box. Impala shares the same Metastore database as Hive.

Standalone Metastore

Hive 4.1+

Since Hive 4.1 the Metastore ships on its own as a binary and a Docker image, with a REST catalog server backed by it, so you can run the catalog without HiveServer2.

See it in a real stack

No architecture uses Apache Hive yet. Drop it into the playground and wire it up yourself.Open playground →

Benchmarks

Benchmark · ClickHouse Inc.ClickBenchHow fast analytical databases run 43 SQL queries over one 100-million-row table of web analytics events, on the same cloud machines.Includes Apache Hive · 7 caveats to read first
How to read a benchmark →
Where it came from2008–2025

By 2008 Facebook was collecting far more data than traditional warehouses could store affordably, so it stored it in Hadoop on commodity hardware. But MapReduce is very low level: every question became a custom program, hard to maintain and reuse.

Facebook’s Data Infrastructure team, including Ashish Thusoo, Joydeep Sen Sarma, Namit Jain and Zheng Shao, built Hive: a SQL-like language compiled into MapReduce jobs, tables mapped to HDFS directories, and a system catalog, the Metastore, holding schemas and statistics. In June 2008 they offered it to Apache as a Hadoop contrib project. Their VLDB 2009 paper reported several thousand tables and more than 700 TB in Facebook’s Hive warehouse, used by over 100 people for reporting and ad-hoc analysis. Hive became a top-level Apache project in September 2010.

Hive made SQL the normal way to query Hadoop. Its batch latency also spawned faster engines over the same tables: Cloudera’s Impala (2012) uses the Metastore, and Facebook built Presto (2013) because Hive on MapReduce was too slow for interactive queries. Hive itself moved to Tez in 2014, and today its Metastore outlives much of the rest of the stack.

Why it’s built this way

SQL compiled to MapReduce
Writing Java MapReduce programs for every report did not scale to a company of analysts. A declarative language let the system plan the jobs, and custom map and reduce scripts could still be plugged in.
Tables are directories, partitions are subdirectories
Data already sat in HDFS. Mapping a table to a directory and a partition such as ds=20090101 to a subdirectory meant no loading step, and the compiler could prune partitions a query didn’t need.
A separate Metastore for schemas and statistics
Files in HDFS carry no schema. Keeping table definitions, partitions and serialization formats in a catalog let the compiler find and parse data, and later let other engines share the same tables.
Pluggable SerDes for file formats
Facebook had data in many formats. Custom serializer and deserializer classes in Java let Hive read any of them without converting the files first.

Timeline

  1. 2008

    Facebook contributes Hive to Hadoop as a contrib project

  2. 2009

    VLDB paper describes Hive at Facebook: 700+ TB, 100+ users

  3. 2010

    Becomes a top-level Apache project in September

  4. 2014

    Hive 0.13 adds Apache Tez as its execution engine

  5. 2016

    Hive 2.0 adds LLAP and deprecates MapReduce

  6. 2024

    Hive 4.0.0 ships; the 1.x, 2.x and 3.x lines reach end of life

  7. 2025

    Hive 4.1 ships a standalone Metastore; 4.2 requires JDK 21

Sources17 · facts checked 6 October 2026

Improve this page on GitHub