Cloud Parts

Ceph

A distributed storage cluster that serves S3 buckets, block devices and a file system from the same pool of disks.

What it is
Object storage,Block storage,File storage
Type
Open source
Made by
Ceph Foundation (Linux Foundation); IBM is the premier sponsor
License
LGPL 2.1 or 3 (core)
Facts checked
5 October 2026
Website Source code

What it is

Ceph is built on RADOS, a cluster of storage daemons that keeps everything as objects in a flat namespace. Each disk runs an OSD daemon, and a small group of monitors keeps the cluster map and agrees on changes to it with Paxos. Clients don’t ask a central server where data lives: they hash an object’s name into a placement group and run CRUSH, a placement function, to map that group onto OSDs according to rules such as “one copy per host” or “one per rack”. Each pool is either replicated (three copies is typical) or erasure-coded, and the OSDs copy, check and repair data among themselves when a disk fails.

Three gateways sit on top. The RADOS Gateway (radosgw, or RGW) is an HTTP server that speaks a large subset of the S3 API, plus OpenStack Swift: it keeps each bucket’s index as key-value data (omap) on RADOS objects in an index pool and stores object contents in a data pool. RBD serves block devices to VMs and Kubernetes volumes, and CephFS adds a POSIX file system with its own metadata servers. New clusters are usually deployed with cephadm, which runs every daemon in a container. On Kubernetes, Rook, a CNCF graduated operator, runs Ceph for you: you declare a CephCluster and a CephObjectStore, Rook creates the pools and RGW pods, and apps can request buckets with an ObjectBucketClaim.

Why teams pick it

  • One cluster serves S3 buckets, block volumes and a shared file system, so a single platform and team can cover all three
  • Broad S3 coverage: versioning, Object Lock, bucket policies, lifecycle rules, server-side encryption and notifications to Kafka, AMQP or HTTP, plus the Swift API
  • CRUSH rules spread copies across hosts, racks or rooms, and the cluster recovers by itself when disks or nodes fail
  • Multisite RGW replicates between zones in different data centres, and every zone can accept writes
  • Mature and widely deployed: research since 2004, a foundation-backed community and commercial support from IBM and others

Watch out for

  • Far more to run than a single S3 server: monitors, managers, OSDs and gateways, each with its own failure modes and upgrade steps
  • Hungry for hardware: monitors need a majority to agree, so plan for at least three hosts, and each OSD targets 4 GB of RAM by default (about 48 GB for a 12-disk node)
  • Performance depends on tuning: the docs say the bucket index and data pools need placement-group tuning beyond the defaults, and a wrong CRUSH rule can put copies in the same failure domain
  • S3 support is a large subset, not all of it: bucket replication works only between RGW zones, and some headers and canned ACLs differ from AWS

Alternatives

What all object storage have in common →
Amazon S3AWSAWS’s object storage, and the de facto API most object stores copy.Apache OpenDALOpen sourceA Rust library, with bindings for Python, Java, Node.js, Go and more, that gives one API (read, write, stat, list, delete) over 50-plus storage backends: S3 and other object stores, local and HDFS file systems, Google Drive and Dropbox, and key-value stores.Apache OzoneOpen sourceA Hadoop-born object store that serves the same buckets over S3 and as a Hadoop file system.Azure Blob StorageAzureAzure’s object storage, with three blob types and an optional real directory tree for analytics.Backblaze B2 Cloud StorageCommercialLow-cost object storage on Backblaze’s own hardware, with its own API and an S3-compatible one.Cloudflare R2CommercialS3-compatible object storage from Cloudflare that charges nothing for data going out.GarageOpen sourceA lightweight S3 object store that keeps copies of your data in several sites linked by ordinary Internet connections.Google Cloud StorageGoogle CloudGoogle Cloud’s object storage, where even the archive class answers in milliseconds.MinIOArchivedOpen sourceA self-hosted object store that speaks the S3 API. The open-source edition is archived and its images are gone from Docker Hub; MinIO now sells AIStor.PGSTY SiloOpen sourceA community-maintained fork of the open-source MinIO server, with builds, images, security fixes and the full web console.RustFSOpen sourceA young Rust reimplementation of MinIO’s design: S3-compatible, erasure-coded, Apache 2.0, and generally available since September 2026.SeaweedFSOpen sourceA distributed blob store with an S3 gateway and a file system layer, built to store and serve billions of small files.TigrisCommercialGlobally distributed S3-compatible object storage that moves data to where it’s read, with no egress fees.Versity S3 GatewayOpen sourceA stateless S3 server that exposes an existing file system, or another object store, through the S3 API.Wasabi Hot Cloud StorageCommercialS3-compatible object storage sold at one flat price per TB, with no egress or request fees.Zenko CloudServerOpen sourceScality’s open-source Node.js S3 server: a light S3 endpoint for development and tests, and a front end that can store objects in other clouds.

How it fits with other technologies

Bootstrap a small cluster with cephadm, add an S3 gateway and use it with the aws CLI · bash

# on the first host: start a monitor and manager in containers
cephadm bootstrap --mon-ip 10.0.0.11
ceph orch host add node2 10.0.0.12
ceph orch host add node3 10.0.0.13
ceph orch apply osd --all-available-devices

# two RADOS Gateway daemons serving S3 on port 8080
ceph orch apply rgw objects --placement="2 node1 node2" --port=8080

# an S3 user; the JSON output contains its access and secret keys
radosgw-admin user create --uid=backups --display-name="Backup jobs"

aws s3 mb s3://backups --endpoint-url http://node1:8080
aws s3 cp --recursive ./exports/ s3://backups/2026-10-05/ --endpoint-url http://node1:8080

Moving from MinIO#

RGW covers nearly everything MinIO users rely on: versioning, Object Lock, bucket policies, lifecycle rules, server-side encryption, notifications and cross-site replication. The difference is the platform underneath. MinIO is one binary per server; Ceph is a cluster with its own operating model, so budget time to learn ceph and radosgw-admin, or let Rook run it if you are already on Kubernetes. Pick Ceph when you also want block or file storage, or need to grow to many petabytes; for buckets alone, a lighter store is less work.

Data moves over the S3 API. Create users with radosgw-admin, recreate buckets and their policies, then copy objects with rclone sync minio:bucket ceph:bucket (rclone has a Ceph provider setting). Run the sync again just before cutover to catch new writes, then switch the endpoint in your apps. Users, policies, lifecycle rules and notification targets don’t travel with the objects and have to be recreated, and a plain sync copies only current versions, not older ones.

See it in a real stack

No architecture uses Ceph yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2004–2023

Ceph began as Sage Weil’s PhD research at the University of California, Santa Cruz, funded by Lawrence Livermore, Sandia and Los Alamos national laboratories, which needed storage that could scale across thousands of disks. Weil wrote the first code in 2004. In November 2006 he presented the Ceph file system at OSDI and CRUSH, its placement algorithm, at SC ’06. RADOS, the object store underneath, followed in 2007, the year he finished his thesis.

Weil kept working on Ceph full-time after graduating, and in 2010 its client was merged into the Linux kernel. In 2012 he founded Inktank to sell support. Red Hat bought Inktank in 2014 for about $175 million; Inktank’s customers already included CERN and Cisco, and many early OpenStack clouds ran on Ceph’s object and block storage.

The Linux Foundation launched the Ceph Foundation in 2018. At the start of 2023 Red Hat’s Ceph team moved to IBM, which became the foundation’s premier sponsor and said its Ceph products would stay 100% open source and upstream-first.

Why it’s built this way

Compute where data lives instead of looking it up
A central table of every object’s location becomes a bottleneck at petabyte scale. With CRUSH, every client and OSD calculates locations from a small cluster map and talks to the right disks directly.
Smart storage daemons
OSDs replicate data, watch their peers and recover lost copies themselves, so the monitors stay small and the cluster heals without a central coordinator.
One object store, several interfaces
Building block devices, a file system and an S3 gateway on the same RADOS layer means one cluster can serve VMs, shared directories and buckets.

Timeline

  1. 2004

    Sage Weil writes the first Ceph code during his PhD at UC Santa Cruz

  2. 2006

    Ceph paper at OSDI and CRUSH paper at SC ’06

  3. 2010

    Ceph client merged into Linux 2.6.34

  4. 2012

    Weil founds Inktank to support Ceph commercially

  5. 2014

    Red Hat acquires Inktank for about $175 million

  6. 2018

    The Linux Foundation launches the Ceph Foundation

  7. 2023

    Red Hat’s Ceph team and product move to IBM

Sources25 · facts checked 5 October 2026

Improve this page on GitHub