Cloud Parts

Object storage

Practically unlimited storage for files of any size, addressed by key.

17 technologies on this page ↓
Live model
What you’re watchingUploads are written as whole objects and viewers read them back by key. Every so often a lifecycle rule moves old objects to the cheap archive tier.

How it works

Object storage keeps blobs (files) in buckets, each under a key like photos/2026/09/a.jpg, with metadata attached. You read and write whole objects through an HTTP API instead of mounting a disk. It scales without capacity planning, replicates across facilities for extreme durability, and offers cheaper tiers for data you rarely touch. A bucket isn’t a disk and has no file system: the folders you see are just keys that contain slashes, and there’s no editing, appending, renaming or locking in place. Tools can mount a bucket as a folder by translating file operations into API calls, with limits. It’s the default home for backups, media, logs and data lakes.

Think of it as

A coat check. Hand over your coat, get a ticket, and use the ticket to get that exact coat back. You can’t alter a sleeve while it’s hanging there.

Reach for it when

  • Images, video, backups, exports and logs
  • Data lakes: Parquet or Iceberg files queried by warehouse engines
  • Static website assets served through a CDN

Think twice when

  • Databases, or anything that rewrites small parts of a file in place
  • Low-latency random reads and writes; that’s block storage
  • Software that needs a real file system (renames, locks, edits in place), unless you mount the bucket with a tool built for it

A real example

A mobile app uploads profile photos straight to a bucket with pre-signed URLs. A lifecycle rule moves photos older than 90 days to an archive tier that costs a fraction as much.

Technologies

Specific products, and what each does differently.

Self-hosted object stores

Software you run on your own disks to serve S3 buckets.

CephOpen sourceA distributed storage cluster that serves S3 buckets, block devices and a file system from the same pool of disks.MinIOArchivedOpen sourceA self-hosted object store that speaks the S3 API. The open-source edition is archived and its images are gone from Docker Hub; MinIO now sells AIStor.PGSTY SiloOpen sourceA community-maintained fork of the open-source MinIO server, with builds, images, security fixes and the full web console.RustFSOpen sourceA young Rust reimplementation of MinIO’s design: S3-compatible, erasure-coded, Apache 2.0, and generally available since September 2026.GarageOpen sourceA lightweight S3 object store that keeps copies of your data in several sites linked by ordinary Internet connections.SeaweedFSOpen sourceA distributed blob store with an S3 gateway and a file system layer, built to store and serve billions of small files.Apache OzoneOpen sourceA Hadoop-born object store that serves the same buckets over S3 and as a Hadoop file system.

What each provider calls it

AWS
S3
Google Cloud
Cloud Storage
Azure
Blob Storage
Open source & others
Ceph (RGW), SeaweedFS, Garage, RustFS, Silo (a MinIO fork), Apache Ozone; MinIO (archived); other clouds: Cloudflare R2, Wasabi, Backblaze B2, Tigris

See it in a real stack

Architectures that use object storage

Small lakehouse on DuckLakeDuckDB writes and queries tables whose catalog lives in PostgreSQL and whose data sits as Parquet on S3.DuckDBDuckLakePostgreSQLAmazon S3Open lakehouse on IcebergKafka streams events into Spark, which writes Iceberg tables on S3 that Trino queries for dashboards.Apache KafkaApache SparkApache IcebergAmazon S3TrinoServerless web app on AWSCloudFront serves the site from S3 and sends API calls through API Gateway to Lambda functions backed by DynamoDB.Amazon CloudFrontAmazon S3Amazon API GatewayAWS LambdaAmazon DynamoDBCQRS with a static read sideCommands update PostgreSQL; a Lambda projector rewrites the affected JSON views in S3, and CloudFront serves every read without touching a database.Amazon API GatewayAWS LambdaPostgreSQLAmazon SQSAmazon S3Amazon CloudFrontBackground jobs with a queueThe API drops slow work onto SQS and returns at once; a pool of Fargate workers drains the queue at a steady pace.AWS LambdaAmazon SQSAWS FargateAmazon S3Code sandboxes for AI agentsAn agent’s code runs in a fresh Firecracker microVM per task, isolated by its own kernel, with outputs saved to S3.FirecrackerAmazon S3IoT telemetry pipelineDevices stream readings into Kinesis; Lambda writes them to TimescaleDB for Grafana, and raw data is archived to S3.Amazon Kinesis Data StreamsAWS LambdaTimescaleDBAmazon S3Grafana

In code · python

import boto3
s3 = boto3.client("s3")

s3.upload_file("a.jpg", "media", "photos/2026/09/a.jpg")

url = s3.generate_presigned_url(          # temporary download link
    "get_object",
    Params={"Bucket": "media", "Key": "photos/2026/09/a.jpg"},
    ExpiresIn=3600,
)

Often confused with

Block storageFile storageCDNsData warehousesBackup & restoreObject, block or file storage?
Where it came from1996–2026

For decades storage came in two shapes: disks of numbered blocks, and file systems that arranged those blocks into directories on a server. As data grew, a file server that every lookup and every byte passed through could become the bottleneck. In 1996 Garth Gibson’s group at Carnegie Mellon proposed Network-Attached Secure Disks (NASD): drives that offered objects instead of bare blocks and let clients read and write them directly, with the file server overseeing access instead of passing every byte through.

The first commercial hit was archival. EMC’s Centera (2002), built on technology from the start-up FilePool, addressed each stored item by a hash of its content and couldn’t modify it afterwards, which suited records that laws such as Sarbanes-Oxley required companies to keep unchanged. Then, in March 2006, Amazon S3 turned the idea into a public web service: buckets of objects under keys, written and read whole over HTTP, billed per GB. Microsoft’s Windows Azure Storage went into production in November 2008, Google previewed its own storage service in May 2010, and that July Rackspace open-sourced the object store behind its Cloud Files as OpenStack Swift.

S3’s API became the de facto standard: MinIO, Cloudflare R2 and Ceph’s object gateway all speak it. Since then the gap with file systems has been narrowing: S3 became strongly consistent in 2020, Azure and Google added optional directory trees, and in 2026 AWS began mounting buckets as NFS file systems.

Why it’s built this way

A flat namespace of keys instead of a directory tree
Without directories to keep consistent, data can be spread across many machines and read in parallel. The trade-off shows up in analytics: renaming a “folder” of 5,000 objects means 10,000 separate copy and delete calls.
Whole objects, never edited in place
Data that is only ever written whole is far easier to replicate, cache and protect. Azure’s engineers wrote that never overwriting committed data “greatly simplified the replication protocol”.
HTTP instead of a mounted device
Any program on any network can reach an object with a URL and credentials, with no driver or mount, which made it natural for the web, mobile apps and data shared between clouds.

Timeline

  1. 1996

    Carnegie Mellon’s NASD project proposes object-based network disks

  2. 1999

    Seagate submits an object storage device command set to the T10 committee

  3. 2002

    EMC Centera ships content-addressed archive storage

  4. 2006

    Amazon S3 launches: buckets, keys and an HTTP API, billed per GB

  5. 2008

    Windows Azure Storage, with blobs, tables and queues, enters production (November)

  6. 2010

    Google previews its storage service (May); Rackspace open-sources OpenStack Swift (July)

  7. 2020

    S3 becomes strongly consistent for every read and list (December)

  8. 2026

    S3 Files mounts buckets as NFS file systems (April)

Sources22

Improve this page on GitHub