Cloud Parts

Apache OpenDAL

A Rust library, with bindings for Python, Java, Node.js, Go and more, that gives one API (read, write, stat, list, delete) over 50-plus storage backends: S3 and other object stores, local and HDFS file systems, Google Drive and Dropbox, and key-value stores.

What it is
Object storage
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
11 October 2026
Website Source code

What it is

OpenDAL (Open Data Access Layer) is a library, not a server or a gateway. You build an Operator from one service’s configuration, such as s3 with a bucket and endpoint, gcs, azblob, fs with a root folder, hdfs, sftp, gdrive or redis, and every call is a method on it: read, write, stat, list, delete, copy, rename. Paths are relative to the operator’s root, and a trailing / marks a directory. Switching from a local folder to S3 means changing the configuration, not the code. Each operator also carries a capability set, because backends differ: the S3 service can presign URLs and a local folder can’t, and a call a service doesn’t support fails with an explicit error instead of silently misbehaving.

Cross-cutting behavior comes from layers that wrap an operator like an onion: retry, timeout, logging, tracing, metrics (Prometheus or OpenTelemetry), throttling, concurrency limits and a hybrid cache (Foyer). Every operation passes through every layer on the way in and out. The core is a Rust crate where each service is a feature flag, so a program compiles in only the backends it enables. The project calls itself “object storage first”: it adds checks real services need, such as treating an S3 CompleteMultipartUpload that returns an error inside a 200 response as a failure, and its S3 service covers S3-compatible stores such as Cloudflare R2, MinIO, Ceph, SeaweedFS, Tigris, Wasabi and Alibaba Cloud OSS.

It began in late 2021 as the storage layer of the Databend warehouse and became a separate project in February 2022. It is an Apache top-level project since January 2024. Users listed by the project include Databend, GreptimeDB, RisingWave, QuestDB, Vector, Mozilla’s sccache, Pants, rustic and the Rust libraries for Apache Iceberg and Apache Paimon. The Rust core is at 0.59.4 (5 October 2026), still 0.x; each language binding has its own version, and the Python package is at 0.47.10.

Why teams pick it

  • One API across very different backends: object stores, file systems, HDFS, consumer drives (Google Drive, OneDrive, Dropbox), protocols (FTP, SFTP, WebDAV, HTTP) and key-value stores (Redis, etcd, TiKV, RocksDB)
  • Production concerns as reusable layers (retry, timeout, logging, tracing, Prometheus and OpenTelemetry metrics, throttling, concurrency limits, caching) instead of per-SDK settings
  • A Rust core with services behind feature flags, so binaries carry only the backends they use, plus bindings for Python, Java, Node.js, Go, Ruby, C, C++, .NET and others
  • Used as the storage layer of real systems (Databend, GreptimeDB, RisingWave, sccache, iceberg-rust) and governed by the Apache Software Foundation rather than one company

Watch out for

  • Still 0.x, and minor versions break the API: 0.59.0 (September 2026) removed direct Metadata constructors, merged OpCopier into OpCopy and changed delete inputs, with an upgrade guide for each release
  • A common API can’t hide every difference: features such as presigning, appends, versions and conditional writes depend on the service’s capabilities, and the project said at graduation that only some of its 59 services were stable
  • Bindings lag and vary: they are versioned separately from the core (Python 0.47.10 against Rust 0.59.4), and the README links published packages only for the Rust, Go, Java, Node.js, Python and Ruby bindings
  • It is a library inside your process, not a service: there is no shared cache, gateway, access control or audit log across applications unless you build them

Alternatives

What all object storage have in common →
Amazon S3AWSAWS’s object storage, and the de facto API most object stores copy.Apache OzoneOpen sourceA Hadoop-born object store that serves the same buckets over S3 and as a Hadoop file system.Azure Blob StorageAzureAzure’s object storage, with three blob types and an optional real directory tree for analytics.Backblaze B2 Cloud StorageCommercialLow-cost object storage on Backblaze’s own hardware, with its own API and an S3-compatible one.CephOpen sourceA distributed storage cluster that serves S3 buckets, block devices and a file system from the same pool of disks.Cloudflare R2CommercialS3-compatible object storage from Cloudflare that charges nothing for data going out.GarageOpen sourceA lightweight S3 object store that keeps copies of your data in several sites linked by ordinary Internet connections.Google Cloud StorageGoogle CloudGoogle Cloud’s object storage, where even the archive class answers in milliseconds.MinIOArchivedOpen sourceA self-hosted object store that speaks the S3 API. The open-source edition is archived and its images are gone from Docker Hub; MinIO now sells AIStor.PGSTY SiloOpen sourceA community-maintained fork of the open-source MinIO server, with builds, images, security fixes and the full web console.RustFSOpen sourceA young Rust reimplementation of MinIO’s design: S3-compatible, erasure-coded, Apache 2.0, and generally available since September 2026.SeaweedFSOpen sourceA distributed blob store with an S3 gateway and a file system layer, built to store and serve billions of small files.TigrisCommercialGlobally distributed S3-compatible object storage that moves data to where it’s read, with no egress fees.Versity S3 GatewayOpen sourceA stateless S3 server that exposes an existing file system, or another object store, through the S3 API.Wasabi Hot Cloud StorageCommercialS3-compatible object storage sold at one flat price per TB, with no egress or request fees.Zenko CloudServerOpen sourceScality’s open-source Node.js S3 server: a light S3 endpoint for development and tests, and a front end that can store objects in other clouds.

How it fits with other technologies

What Apache OpenDAL uses

The same code against a local folder and S3 (MinIO here), with a retry layer · python

# pip install opendal   (Python binding 0.47.10)
import opendal
from opendal.layers import RetryLayer

def save_report(op):
    op.write("reports/2026-10.csv", b"day,orders\n1,42\n")
    print(op.stat("reports/2026-10.csv").content_length, "bytes")
    print([e.path for e in op.list("reports/")])
    print("presign:", op.capability().presign)

save_report(opendal.Operator("fs", root="/tmp/odal-demo"))
# 16 bytes
# ['reports/', 'reports/2026-10.csv']
# presign: False

s3 = opendal.Operator(
    "s3",
    bucket="demo",
    region="us-east-1",
    endpoint="http://localhost:19000",
    access_key_id="minioadmin",
    secret_access_key="minioadmin",
).layer(RetryLayer(max_times=3))
save_report(s3)
# 16 bytes
# ['reports/2026-10.csv']
# presign: True
print(s3.read("reports/2026-10.csv").decode())
# day,orders
# 1,42

See it in a real stack

No architecture uses Apache OpenDAL yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2021–2026

OpenDAL started inside Databend, a cloud data warehouse written in Rust. In late 2021 Xuanwo began redesigning and reimplementing its storage access layer: on 21 December 2021 a pull request titled “dal2: Add basic operations of read, write, stat, delete” opened in the Databend repository, a “Vision of Databend DAL” proposal followed on 27 December, and dal2 was merged into Databend on 29 December.

On 14 February 2022 dal2 left the Databend repository as its own project, renamed OpenDAL. Other Rust data systems adopted it: GreptimeDB was the first large Rust database after Databend, Mozilla’s sccache began using it for its storage layer, OpenDAL soon added a GitHub Actions cache service, and RisingWave and Vector followed. By then it supported S3, Azure Blob, GCS and HDFS. By March 2023 it had released 99 versions and had 48 developers.

Databend’s creators chose to contribute the project to the Apache Software Foundation. Its incubation was approved on 27 February 2023, and the code moved to the ASF on 15 March 2023. It graduated as a top-level project on 18 January 2024, with 19 committers and 59 services, and the community said it would now focus on making existing services stable and on documentation for the non-Rust bindings rather than on adding services.

Why it’s built this way

Object storage first
Databend’s data lived in S3-style stores, so the design favors them: its buffers are tuned for HTTP-based services to avoid extra copies, and it adds checks those services need even when they cost a little speed.
Describe each service with capabilities
Backends differ too much for a lowest common denominator. Capabilities let OpenDAL use each service’s native features where they exist and fail loudly where they don’t.
Layers around one operator
Retry, logging, metrics and timeouts are needed for every backend. Writing them once as stackable layers keeps the core free of any single policy and lets users add their own.
One Rust core, many thin bindings
Implementing each service once in Rust and binding it into other languages gives Python, Java or Node.js users the same behavior, at the cost of bindings that are released on their own schedules.

Timeline

  1. 2021

    dal2, a new storage access layer, is written for Databend (December)

  2. 2022

    Leaves the Databend repository as OpenDAL, February 14

  3. 2023

    Enters the Apache Incubator (approved February 27, code transferred March 15)

  4. 2024

    Graduates as an Apache top-level project, January 18

  5. 2026

    Release 0.59 adds conditional deletes and S3 Express auth, with breaking API changes (September)

Sources13 · facts checked 11 October 2026

Improve this page on GitHub