Cloud Parts

Apache Hadoop

The project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.

What it is
Query engines,File storage
Used for
Data lakes
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
6 October 2026
Website Source code

What it is

Hadoop is three systems in one Java project. HDFS stores files as large blocks, 128 MB by default, on the disks of many DataNodes and keeps three copies of each block, at least one on another rack. A NameNode holds the whole namespace and the map of where every block lives in memory; clients ask it where the blocks are, then read and write directly to the DataNodes. Files are written once and can only be appended to or truncated later. Since Hadoop 2.0 a standby NameNode, kept in sync through a quorum of JournalNodes, can take over when the active one fails. YARN runs the work: a ResourceManager hands out containers (slices of CPU and memory) on the NodeManagers that run on every machine, and each application gets its own ApplicationMaster to request containers and track its tasks. MapReduce is one such application: map tasks run where the blocks are, their output is sorted and shuffled by key to reduce tasks, and the results go back to HDFS.

Hadoop’s file system API also reaches object stores: the S3A connector for S3 and compatible stores, ABFS for Azure Data Lake Storage Gen2 and, since 3.5, Google Cloud Storage. Spark and other engines use these connectors even on clusters without HDFS or YARN. Hadoop is free under Apache 2.0; the current release is 3.5.0 (April 2026), which requires Java 17 on servers. The commercial distributions have consolidated: Cloudera and Hortonworks merged in January 2019, and Cloudera was taken private by Clayton, Dubilier & Rice and KKR in October 2021. In the cloud, Amazon EMR runs Hadoop and Spark on clusters where HDFS is temporary and data lives in S3, and Google now calls Dataproc “Managed Service for Apache Spark”.

Why teams pick it

  • Proven at very large scale on commodity servers, with rack-aware replication or erasure coding to survive disk, node and rack failures
  • Moves computation to the data: tasks are scheduled on the machines that hold their blocks, so big scans don’t cross the network
  • HDFS gives real file system semantics, including atomic renames, which commit protocols rely on and object stores lack
  • YARN lets MapReduce, Spark and other engines share one cluster with queues and capacity limits
  • Its connectors (S3A, ABFS, GCS) are how Spark and many other JVM tools read and write object storage

Watch out for

  • Heavy to run: NameNodes, JournalNodes, DataNodes, ResourceManagers and NodeManagers, usually with Kerberos, each needing tuning and upgrades
  • Storage and compute scale together: adding disk means adding servers, and data stored three times costs 200% overhead unless you use erasure coding
  • The NameNode keeps every file and block in memory, so millions of small files exhaust its heap long before the disks fill
  • MapReduce writes to disk between every step, and higher-level tools have left it: Hive deprecated its MapReduce engine in Hive 2.0
  • Commercial distributions have consolidated: Hortonworks merged into Cloudera, now privately held, and Google renamed its managed Hadoop and Spark service after Spark alone

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

What uses Apache Hadoop

Load files into HDFS, run a MapReduce job on YARN, copy the result to S3 · bash

# put a log file into HDFS and see how its blocks were placed
hdfs dfs -mkdir -p /data/logs
hdfs dfs -put access-2026-10-05.log /data/logs/
hdfs fsck /data/logs/access-2026-10-05.log -files -blocks -locations

# run the bundled word-count example as a MapReduce job on YARN
yarn jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar \
  wordcount /data/logs /data/wordcount
hdfs dfs -cat /data/wordcount/part-r-00000 | sort -k2 -nr | head

# copy the output to S3 with a parallel DistCp job (S3A connector)
hadoop distcp /data/wordcount s3a://acme-archive/wordcount/2026-10-05/

HDFS or object storage?#

In the cloud, data usually stays in a bucket and jobs reach it through Hadoop’s connectors; on Amazon EMR, HDFS is only scratch space. The trade-offs, as Hadoop’s and Spark’s own docs describe them:

HDFSS3 through S3A
Where data livesDisks inside the cluster’s DataNodesA bucket run by the provider, outside any cluster
MetadataOne active NameNode, with the whole namespace in memoryRun by the provider; directories are emulated from key prefixes
Directory renameAtomic, as Hadoop’s file system spec requiresNot atomic; time grows with the number and size of files
Reading dataTasks run on the machines that hold the blocksEvery read crosses the network; seeks can need new HTTP requests
On Amazon EMRTemporary: reclaimed when the cluster endsPersistent: outlives the cluster
CapacityAdd servers; three replicas by default, or erasure coding (Reed-Solomon 10,4 stores 1.4x)Grows without adding servers; billed per GB stored and per request
  • Because renames on S3 aren’t atomic, Spark and MapReduce jobs writing to S3 should use the S3A committers rather than the classic rename-based commit.

What happened to each piece#

MapReduce

still ships

Kept for compatibility, but higher-level tools moved off it: Hive deprecated its MapReduce engine in Hive 2.0, and Spark keeps data in memory between steps instead of writing every stage to disk.

YARN

on-premises clusters

Still schedules Spark and MapReduce on Hadoop clusters. Spark can also run on Kubernetes or its own standalone cluster manager, with no YARN at all.

HDFS

replaced in the cloud

Still the file system of on-premises clusters. Apache Ozone grew out of the Hadoop project to scale past the NameNode with an object store; in the cloud, buckets hold the persistent data.

Hadoop client libraries

everywhere

The S3A, ABFS and GCS connectors and the FileSystem API are how Spark and many other JVM tools read and write object storage, even with no Hadoop cluster in sight.

Distributions

consolidated

Hortonworks was founded in 2011 by Yahoo’s Hadoop engineers and merged into Cloudera in 2019. Amazon EMR and Google’s managed Spark service still run Hadoop components on demand.

See it in a real stack

No architecture uses Apache Hadoop yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2003–2026

In 2003 and 2004 Google described how it stored and processed the web on thousands of cheap machines: the Google File System, and MapReduce, a model where you write a map and a reduce function and the framework handles splitting, scheduling and failures. Nutch, an open-source web search engine, needed exactly that to crawl and index the web, so its developers built a distributed file system modelled on GFS and their own implementation of MapReduce.

In February 2006 Doug Cutting moved that code out of Nutch into a new Lucene sub-project named Hadoop. Yahoo’s grid team started using and extending it the next month, and took it from dozens of machines to thousands: by February 2008 Yahoo’s Search Webmap ran on more than 10,000 cores and fed every Yahoo web search. Hadoop became a top-level Apache project in January 2008.

Hadoop made storing and processing petabytes on commodity servers routine and grew an ecosystem of tools and companies, including Cloudera and Hortonworks. Hadoop 2 added YARN so engines other than MapReduce could share a cluster. Since then Spark and cloud object storage have taken over much of what Hadoop clusters did, and Cloudera and Hortonworks merged in 2019.

Why it’s built this way

Large blocks, written once, copied three times
HDFS assumed huge files read in long streams on hardware where failure is the norm. Big replicated blocks and write-once files (append and truncate only) kept the design simple and survived lost disks and nodes.
Move the computation to the data
Shipping a task to the machine that holds its block is cheaper than pulling terabytes across the network, so the scheduler places map tasks next to their input.
One NameNode holding all metadata in memory
A single master made the namespace simple and fast, but it was a single point of failure until Hadoop 2.0 added standby NameNodes, and it still caps the number of files.
Splitting scheduling out of MapReduce (YARN)
Hadoop 1 had one JobTracker that queued and ran all MapReduce jobs. YARN separated cluster resource management from per-job scheduling, so other frameworks could run on the same machines.

Timeline

  1. 2003

    Google publishes the GFS paper; the MapReduce paper follows in 2004

  2. 2006

    NDFS and MapReduce leave Nutch as Hadoop; Yahoo adopts it

  3. 2008

    Top-level Apache project; Yahoo’s Webmap runs on 10,000+ cores

  4. 2011

    Hadoop 1.0.0 released in December

  5. 2013

    Hadoop 2.2.0 GA brings YARN and HDFS high availability

  6. 2017

    Hadoop 3.0 adds HDFS erasure coding

  7. 2026

    Hadoop 3.5.0 requires Java 17 and adds a GCS connector

Sources27 · facts checked 6 October 2026

Improve this page on GitHub