Cloud Parts

Apache AmoroIncubating

An incubating Apache “lakehouse management system”, started at NetEase as Arctic, that watches Iceberg tables and keeps them fast on its own: it compacts small files, rewrites deletes and expires old snapshots and files in the background.

What it is
Lakehouses,Table formats
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Status
Incubating. In the Apache Incubator since March 2024, so not yet a top-level Apache project.
Facts checked
11 October 2026
Website Source code

What it is

Amoro has two parts. AMS, the Amoro Management Service, is a central server that tracks every table it manages, notices when files pile up and plans optimizing tasks, then commits the results; it also serves a web dashboard, a catalog service and a Spark SQL terminal. Optimizers are separate, stateless worker processes that pull those tasks and execute them. They run as a local Java process, as Kubernetes pods, as a Flink or Spark job, or as anything you start yourself, and they are pooled into optimizer groups so one group of tables can’t starve another. AMS can be the metastore itself (an “internal catalog”, which it also serves as an Iceberg REST catalog) or manage tables that live in Hive Metastore, a Hadoop catalog, AWS Glue, a REST catalog or Nessie.

Its “self-optimizing” borrows from generational garbage collection. Files under one eighth of self-optimizing.target-size (128 MB by default, so 16 MB) are fragments; larger ones are segments. Minor optimizing runs often, merging fragments and turning equality deletes into position deletes; by default it triggers at 12 fragment and delete files or after an hour. Major optimizing runs rarely and merges segments to drop rows that deletes and duplicate keys have made redundant, and full optimizing rewrites everything into one file per space. Besides plain Iceberg tables, Amoro has its own formats for primary-key tables fed by CDC streams: Mixed-Iceberg, built from Iceberg tables underneath, and Mixed-Hive, which Hive can still read and write. It also shows Paimon and Hudi tables’ metadata. It has been in the Apache Incubator since March 2024. The latest release is 0.8.1-incubating (11 September 2025); 0.9.0 has been in release candidates since September 2026, and the Docker image tagged latest already points at one.

Why teams pick it

  • Maintenance runs continuously without anyone scheduling Spark jobs: AMS watches commits and plans compaction, delete conversion, snapshot expiry and orphan-file cleanup per table
  • Minor and major optimizing are tuned separately, so streaming tables get small files merged quickly without rewriting large files over and over
  • Optimizers run where you already have capacity (Flink, Spark, Kubernetes or a plain JVM), and optimizer groups with Quota or Balanced scheduling keep tables from competing for the same workers
  • Works with existing catalogs: register Hive Metastore, Glue, a REST catalog or Nessie and Amoro maintains the tables in place, with no Amoro dependency for engines that read Iceberg tables

Watch out for

  • Still incubating, with a slow release cadence: 0.8.1 from September 2025 is the latest release, and 0.9.0 had reached its twelfth release candidate by September 2026
  • Lags Iceberg’s newest format: 0.8 bundles Iceberg 1.6.1, and the 0.9 docs say format v3 tables are skipped for optimizing and maintenance because the bundled writer could make them unreadable to v3 readers
  • Another service to run: AMS keeps its state in embedded Derby unless you give it MySQL or PostgreSQL, and optimizers need a Flink, Spark or Kubernetes cluster to scale beyond the local container, which the docs recommend only for demos
  • Mixed-Iceberg and Mixed-Hive tables are Amoro’s own formats: their full features need Amoro’s Flink, Spark and Trino connectors, and the docs warn that primary-key uniqueness or conflict resolution can fail where tables are also used through Hive or plain Iceberg

Alternatives

What all lakehouses have in common →
Amazon S3 TablesAWSA bucket type for Apache Iceberg tables: S3 stores them, acts as their catalog, and compacts and cleans them up for you.DatabricksCommercialA managed lakehouse platform built on Apache Spark and Delta Lake, with Unity Catalog for governance, on AWS, Azure or Google Cloud.DremioCommercialA SQL lakehouse engine built on Apache Arrow, with its own Iceberg catalog and “reflections” that speed up queries without anyone changing them.SpiralCommercialA closed-source storage and query engine for multimodal AI data (video, images, sensor traces and tables) that keeps everything as Vortex files in your own object storage, versions every change and streams results to GPUs; access is by request.Tiger LakeBetaCommercialTiger Cloud’s built-in Iceberg connector, in public beta: it copies a Postgres table or TimescaleDB hypertable into Apache Iceberg, then keeps it current from the write-ahead log.

How it fits with other technologies

Twelve tiny appends through AMS’s Iceberg REST endpoint, then let self-optimizing merge them · python

# pip install "pyiceberg[pyarrow]"   (pyiceberg 0.12)
# AMS 0.8.1 (apache/amoro:0.8.1-incubating) at amoro:1630, with an internal Iceberg
# catalog `lake` whose optimizer group `local` has one running local optimizer
import time
import pyarrow as pa
from pyiceberg.catalog.rest import RestCatalog

# AMS serves an Iceberg REST catalog for its internal catalogs
catalog = RestCatalog("amoro", uri="http://amoro:1630/api/iceberg/rest", warehouse="lake")
catalog.create_namespace("db")
events = catalog.create_table(
    "db.events",
    pa.schema([("id", pa.int64()), ("kind", pa.string())]),
    properties={"self-optimizing.minor.trigger.file-count": "10"},  # default 12
)

# A streaming-style writer: twelve tiny commits, one small Parquet file each
for i in range(12):
    events.append(pa.table({"id": [i], "kind": ["click"]}))

def data_files():
    return len(catalog.load_table("db.events").inspect.files())

print(data_files())  # 12

time.sleep(120)       # AMS notices the fragments and plans a minor optimizing

t = catalog.load_table("db.events")
print(data_files(), t.current_snapshot().summary.operation)  # 1 Operation.REPLACE
print(t.scan().to_arrow().num_rows)                          # 12

See it in a real stack

No architecture uses Apache Amoro yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2019–2026

NetEase began using Apache Iceberg across several business lines in 2019. Tables written by streams and CDC piled up small files and delete files, and someone had to run compaction and cleanup jobs at the right pace: too rarely and reads slowed sharply, too often and the rewrites wasted resources. NetEase built a management system called Arctic to do that work automatically.

NetEase open-sourced Arctic in 2022 and renamed it Amoro in 2023; the first release under the new name, 0.5.0, came out in August 2023. By the time of its Apache Incubator proposal it supported more table formats, had 62 developers, had shipped eight versions in a year and was used at more than ten other companies. Jinsong Zhou of NetEase is listed as the project’s founder.

Amoro entered the Apache Incubator on 11 March 2024, with initial committers from NetEase, DtDream, Cisco Webex and Qichacha. The first Apache release, 0.7.0-incubating, followed in August 2024, and 0.8.0 in May 2025 moved to Iceberg 1.6.1, added Spark 3.5 for Mixed tables and showed Hudi metadata in the dashboard.

Why it’s built this way

Minor and major optimizing, after generational GC
Optimizing often keeps reads fast but rewrites the same data again and again. Splitting files into fragments and segments, as a JVM splits young and old objects, lets small files merge quickly while big ones are rewritten rarely.
A central planner with separate, stateless optimizers
The proposal describes AMS as the one place that watches every table and schedules work, while optimizers carry no state, so they can be deployed at scale on Flink or Spark and grouped to isolate resources.
Table formats as pluggable storage engines
The docs compare it to MySQL and ClickHouse choosing storage engines: Amoro manages Iceberg, its Mixed formats and Paimon side by side rather than defining one new format.
Mixed-Hive tables built on existing Hive tables
Many users didn’t want to disturb jobs built on Hive, so a Hive table can be upgraded by migrating metadata only, and Hive keeps reading and writing it as before.

Timeline

  1. 2019

    NetEase starts Arctic to manage its Iceberg tables

  2. 2022

    Arctic is open-sourced; 0.3.0 is released in July

  3. 2023

    Renamed Amoro; 0.5.0 is the first release under the new name, in August

  4. 2024

    Enters the Apache Incubator on 11 March; 0.7.0-incubating in August

  5. 2025

    0.8.0 in May moves to Iceberg 1.6.1; 0.8.1 follows in September

  6. 2026

    0.9.0 release candidates begin in September

Sources15 · facts checked 11 October 2026

Improve this page on GitHub