cloudparts

Open Data FabricExperimental

An open spec for datasets that keep their full history: Parquet files plus a hash-linked chain of metadata blocks, derived data made by deterministic streaming queries, and protocols for sharing it all.

What it is
Table formats
Used for
Stream processing
Type
Open source
Made by
Kamu Data, Inc.
License
Apache 2.0 (the specification and schemas)
Status
Experimental. The spec is at version 0.42 and still changes through RFCs; Kamu, its reference implementation, lists the dataset layout as unstable.
Facts checked
23 October 2026
Website Source code

What it is

Open Data Fabric (ODF) defines a dataset as a stream of events that only grows. The data is stored as Parquet files and read into Apache Arrow. Next to it sits the metadata chain: a list of blocks, each holding the SHA3-256 hash of the previous one, that records the dataset’s identity, its sources, schema, each slice of data added (with offsets and file hashes), engine checkpoints and every transform that ran. Blocks, data files and checkpoints are all named by the hash of their content, so the hash of the last block acts as a Merkle root: anyone holding it can check a copy of the whole dataset fetched from anywhere. Mistakes are fixed by appending retractions and corrections, not by rewriting files.

Root datasets take in outside data; derivative datasets are defined only as queries over other datasets. Those queries run in engines, such as Flink, Spark or DataFusion behind a gRPC contract, inside sandboxed OCI containers with no network access, so the same inputs and engine image always give the same output and anyone can replay a pipeline to verify it. The storage layout needs nothing but reading objects by key, so a dataset can live on a local disk, in S3 or on IPFS, and simple and smart transfer protocols copy only new blocks and files. Datasets are identified by did:odf DIDs. Kamu is the reference implementation; the spec is maintained in the open with RFCs and a consortium of companies and universities.

Why teams pick it

  • Provenance is built into the format: every block names its inputs, query, engine image and output hashes, so any derived number can be traced back to its sources
  • Verifiable without trusting the host: a dataset copied from anywhere can be checked against one trusted block hash, and derived datasets can be checked by replaying their transforms
  • Keeps full history by default, with explicit retractions and corrections, so past states stay queryable and fixes flow through downstream pipelines
  • Built from open pieces: Parquet, Arrow, FlatBuffers, multihash and W3C DIDs, with a storage layout that works on plain object storage or content-addressed networks

Watch out for

  • Kamu is the only implementation the project points to, so the open format doesn’t yet give you a choice of engines and tools the way Iceberg or Delta Lake do
  • Pre-1.0 and evolving through RFCs; parts of the spec, such as determinism enforcement, engine versioning, dataset ownership and signing, are not implemented yet
  • Deterministic transforms rule out calling APIs or using libraries outside the engine, and keeping all history costs storage unless you compact or prune
  • Data in permanent networks such as IPFS or Arweave can’t be reliably deleted, which the spec warns matters for erasure and unencrypted data

Specs

Compare with alternatives →
Data model
Append-only event datasets with a hash-linked chain of metadata blocks; fixes are retractions and corrections
Storage format
Parquet data files and FlatBuffers metadata blocks, named by content hash, on disk, S3 or IPFS

Alternatives

What all table formats have in common →Compare side by side →
Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.LakeSoulOpen sourceA lakehouse table format, first built by DMetaSoul, that keeps all metadata and commits in PostgreSQL and stores primary-key tables as hash buckets with merge-on-read upserts, aimed at CDC ingestion and frequent updates.LanceOpen sourceA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.

How it fits with other technologies

How a dataset is laid out in a file system or object store · text

{dataset-root}/
├── blocks/
│   └── {block-hash}             # metadata blocks, each linking to the previous one
├── refs/
│   └── head                     # hash of the latest block: the Merkle root
├── data/
│   ├── {data-file-physical-hash}  # Parquet files
│   └── {linked-object-hash}       # large files linked from data columns
└── checkpoints/
    └── {checkpoint-physical-hash} # saved engine state

What the spec covers#

The specification is split into groups that build on each other.

Datasets

Logical structure, identity, the event data model, schemas, the metadata chain, data and checkpoint hashing, storage layout and ingestion.

Transformations

The streaming processing model and the engine contract: how a coordinator runs a sandboxed engine and records the result.

Exchange

Simple and smart transfer protocols for copying datasets, and verifiable batch queries that come with a commitment to their result.

Automation

An infrastructure-as-code resource framework and a task and flow system (RFC-018 and RFC-019, 2026).

Auth and privacy

An authorization model for who may read and change datasets (RFC-020, 2026).

See it in a real stack

No architecture uses Open Data Fabric yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2020–2026

Data shared between organisations usually travels as snapshots: an export from a database, a CSV on a portal, a copy in someone’s warehouse. Each copy loses the history between snapshots and any link to where it came from, and when someone cleans or corrects it the work stays in their copy. Sergii Mikhtoniuk and Özge Nilay Yalçın, who founded Kamu, set out to design data exchange where every dataset carries its full history and lineage and anyone can check it without trusting a central party.

After about a year of prototyping, they wrote the Open Data Fabric specification and a whitepaper in 2020, published the spec in the open, and built the kamu CLI as its reference implementation. The whitepaper was rejected by several data conferences and went to arXiv in November 2021. In 2021 a consortium with Dell, Intel and the University of Groningen formed around it.

The spec has grown through numbered RFCs: record offsets, logical data hashes, content addressability for storage networks like IPFS, push sources, a unified changelog schema, and in 2026 an infrastructure-as-code resource framework, a task and flow system and an authorization model. The current version is 0.42.1.

Why it’s built this way

Event streams instead of tables
Snapshots of tables lose what happened between them. Treating data as an append-only stream of events keeps full history and lets one query run forever over new data.
A hash-linked metadata chain
Linking every block to the previous one by hash makes the metadata tamper-evident and lets a whole dataset be checked from one trusted hash, much like Git history or a blockchain.
Engines as sandboxed containers
Writing a new engine was not the goal. Running existing engines in OCI containers with no network access, and recording the image digest, keeps transforms deterministic and replayable.
Parquet and Arrow for the data
Parquet is a widely supported, compact columnar format that decodes quickly into Arrow, the in-memory format engines can share without conversion.

Timeline

  1. 2020

    Spec and whitepaper written; spec repository published in April

  2. 2021

    Whitepaper on arXiv in November; consortium with Dell, Intel and the University of Groningen

  3. 2022

    Kamu adds IPFS storage and joins Protocol Labs’ Compute Over Data group

  4. 2026

    RFC-018 to RFC-020: IaC resources, tasks and flows, authorization

  5. 2026

    Spec version 0.42.1

Sources10 · facts checked 23 October 2026

Improve this page on GitHub