Cloud Parts

Spiral

A closed-source storage and query engine for multimodal AI data (video, images, sensor traces and tables) that keeps everything as Vortex files in your own object storage, versions every change and streams results to GPUs; access is by request.

What it is
Lakehouses,Query engines
Type
Commercial product
Made by
SpiralDB (Spiral)
License
Proprietary (the Python SDK is published under a proprietary licence); its storage format, Vortex, is Apache 2.0
Facts checked
6 October 2026
Website

What it is

Spiral comes from the team that created the Vortex file format. Its pitch is that AI training is a new kind of data consumer: a GPU wants millions of samples per second, random reads across a whole dataset, and objects between about 1 KB and 25 MB (embeddings, small images, documents) that neither Parquet tables nor one-object-per-file storage handle well. Spiral stores data as Vortex in a bucket you already have (S3, Google Cloud Storage, R2, MinIO or Tigris), queries formats such as MCAP robot logs, LeRobot datasets, MP4 video and Parquet in place, and loads query results onto GPUs for PyTorch without a separate export step. A query can be re-run as of an exact commit, tag or branch, and Spiral is rolling out a “journal” that records every change to shared data as a transaction.

Little of this can be checked from outside. The engine is closed source, the documentation sits behind a login, and teams get in by request: Spiral says it onboards “a few teams at a time”. What is public is a Python SDK on PyPI (spiraldb, proprietary licence) that also exposes an embedded engine as an ADBC driver and connects to a hosted catalog, plus engineering posts on the company blog. Its positioning has also moved. The September 2025 launch described a general data platform for AI with unified permissions; in 2026 the site focuses on curating training data for robotics and other physical AI.

Why teams pick it

  • Data stays in your own bucket, stored in Vortex, an open format with Apache 2.0 readers, rather than inside a proprietary store
  • One system for tabular data, video, images and sensor traces, aimed at building and versioning training datasets
  • Versioned history: query results can be tied to an exact commit, tag or branch and re-run later
  • Built by the Vortex authors, who publish detailed posts on compression and on moving data from S3 to GPUs

Watch out for

  • Closed source with a proprietary SDK; the docs are not public and access is by request
  • Early: Spiral describes working with design partners, and the Python SDK has already split into a legacy package (pyspiral) and a new one (spiraldb)
  • Throughput and speed claims on its site come from the vendor and its customers, with no independent benchmarks
  • The product focus has shifted from general AI data infrastructure (2025) to robotics and physical AI datasets (2026)

Alternatives

What all lakehouses have in common →
Amazon S3 TablesAWSA bucket type for Apache Iceberg tables: S3 stores them, acts as their catalog, and compacts and cleans them up for you.Apache AmoroIncubatingOpen sourceAn incubating Apache “lakehouse management system”, started at NetEase as Arctic, that watches Iceberg tables and keeps them fast on its own: it compacts small files, rewrites deletes and expires old snapshots and files in the background.DatabricksCommercialA managed lakehouse platform built on Apache Spark and Delta Lake, with Unity Catalog for governance, on AWS, Azure or Google Cloud.DremioCommercialA SQL lakehouse engine built on Apache Arrow, with its own Iceberg catalog and “reflections” that speed up queries without anyone changing them.Tiger LakeBetaCommercialTiger Cloud’s built-in Iceberg connector, in public beta: it copies a Postgres table or TimescaleDB hypertable into Apache Iceberg, then keeps it current from the write-ahead log.

How it fits with other technologies

SQL through the SDK’s embedded engine (from the spiraldb package README) · python

# pip install spiraldb
from spiraldb.adbc import dbapi

with dbapi.connect() as connection:
    with connection.cursor() as cursor:
        cursor.execute("SELECT CAST(? AS BIGINT) + 1 AS value", [41])
        assert cursor.fetchall() == [(42,)]

What Spiral says the product does#

These are the vendor’s own descriptions from its homepage, not independently verified.

Multimodal engine

claim

Queries across video, audio, images, traces and tables at any scale.

Your storage

S3, GCS, R2, MinIO, Tigris

Reads from and writes to the bucket you already have, as Vortex files.

In-place queries

MCAP, LeRobot, MP4, Parquet

Queries existing files without ingesting them first.

GPU data loading

CUDA, NVDEC, PyTorch

Streams query results into training loops on GPUs, without staging a copy.

Time travel

commit, tag, branch

Re-runs a query against the exact version of the data behind an earlier result.

See it in a real stack

No architecture uses Spiral yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2024–2026

SpiralDB’s founders set out to build an object-store native data system for AI. In their telling, databases first served humans typing in records, then “big data” systems turned machine-collected events into dashboards for humans, and now machines are the consumers too: an NVIDIA H100 has the memory bandwidth to take in 4 million 100 KiB images a second, and a Monte Carlo tree search may need billions of random reads. Parquet and object storage both cope badly with objects between about 1 KB and 25 MB, so AI teams read Parquet, expand it to Arrow, convert it to tensors and cache the results before they can train.

Their first step was storage. They built the Vortex columnar format and in August 2025 donated it to the Linux Foundation. In September 2025 Spiral came out of stealth with $22 million in seed and Series A funding from Amplify Partners and General Catalyst. CEO Will Manning described Spiral as the data platform on top of Vortex: object-store native, with one permission model across data types and one API for embeddings, images and video, working with design partners in computer vision, robotics and multimodal AI.

Through 2026 the company narrowed its pitch to training data for robotics and physical AI. In August 2026 it described a “journal” that records every change to shared data as a transaction, so past states stay addressable and experiments can be rolled back, and said it was deploying it first with design partners. In September 2026 it described cutting the copies on the path from S3 to GPU memory down to one.

Why it’s built this way

Object storage as the system of record
Data lives in cheap object storage owned by the customer, with caches added only where access needs speeding up, and users bring their own compute, from DataFusion, DuckDB and Trino to PyTorch, Spark and Ray.
Store each object according to its size
Inlining large blobs in a table kills scan performance, and storing pointers breaks governance. Spiral says it keeps 10 KB embeddings inside Vortex, batches 10 MB images into blocks sized for S3, and leaves 4 GB videos where they are without copying them.
A journal of transactions, not copies
Table formats version rows but not the external blobs (MCAP, video, calibration files) they point to, so rewriting a file breaks old snapshots. Making every change a transaction keeps every past state readable and lets experiments, or agents, be rolled back.
One copy from S3 to the GPU
Each copy along the network path costs CPU per gigabyte. The team wrote its own HTTP client that reads straight into pinned buffers and moved TLS decryption into the Linux kernel (kTLS), so moving each gigabyte to the GPU takes less CPU.

Timeline

  1. 2024

    Work on Vortex, Spiral’s storage format, appears on GitHub (February)

  2. 2025

    Vortex donated to the Linux Foundation’s LF AI & Data (August)

  3. 2025

    Spiral launches with $22M in seed and Series A funding (September)

  4. 2026

    The journal, for versioned changes to shared data, goes to design partners (August)

  5. 2026

    Engineering post on moving data from S3 to GPU memory in one copy (September)

Sources8 · facts checked 6 October 2026

Improve this page on GitHub