Cloud Parts

Vortex

A columnar file format and toolkit that keeps data compressed with modern encodings, reads single rows fast and runs filters directly on the compressed data, pitched as a faster successor to Parquet.

What it is
File formats (columnar files)
Type
Open source
Made by
LF AI & Data (Linux Foundation), started at SpiralDB
License
Apache 2.0
Facts checked
6 October 2026
Website Source code

What it is

Parquet encodes each column and then usually squeezes every page with a general-purpose codec such as Snappy or Zstd, so a reader has to decompress a whole page before it can filter it or fetch one row. Vortex stores columns with lightweight encodings from recent database research instead: FastLanes for integers, FSST for strings, ALP for floating-point numbers, plus dictionary and run-length encoding, layered on top of each other by a sampling compressor modelled on BtrBlocks. Many of these encodings can be read at a single position, and Vortex’s compute functions can work on them directly. A comparison on dictionary-encoded data, for example, runs once over the small set of unique values instead of every row. In memory, every encoded array can be converted to a canonical form, which for almost every type matches Apache Arrow.

A file is a tree of layouts: struct layouts split a table into columns, chunked layouts split columns into row ranges, zone maps store statistics for blocks of rows, and flat layouts hold the encoded data in segments. The footer is small FlatBuffer metadata, so a reader can find one column among 100,000 without parsing the rest. The project claims 100× faster random access, 10 to 20× faster scans and 5× faster writes than Parquet at similar compression; those are its own benchmarks. Readers and writers exist for Rust, Python, Java and C/C++, with integrations for DataFusion, DuckDB, Spark, Trino, Polars and Ray. The library is still pre-1.0 (0.87 in October 2026), but the file format has been stable since 0.36.0 in May 2025: newer readers can read any file written since then.

Why teams pick it

  • Fast reads of individual rows by position, so lookups and shuffled sampling don’t require reading whole pages
  • Filters and other compute can run on compressed data, saving decompression work and memory bandwidth
  • Encodings, layouts and compression strategies are plug-ins, so new compression research can be added without a new format
  • Zero-copy conversion to and from Apache Arrow, and native readers in DataFusion, DuckDB and Spark
  • Neutral governance: an Apache 2.0 project in the Linux Foundation’s LF AI & Data, not owned by one company

Watch out for

  • Young: library APIs still change between releases, and far fewer engines and warehouses read it than Parquet
  • Table-format support is limited; the project lists Apache Iceberg integration as “coming soon”
  • Small files carry a lot of overhead: the docs’ 71 KiB example is twice the size of the Parquet original, and Vortex only matches Parquet from tens of megabytes up
  • The headline speedups come from the project’s own benchmarks, and SpiralDB, which started it, builds a commercial product on it
  • Newer readers can read old files, but a writer using new encodings has to target an older “edition” for older readers to keep up

Alternatives

What all columnar files have in common →
Apache ORCOpen sourceA columnar file format born in Hive: typed columns in large stripes, built-in indexes every 10,000 rows, and the storage behind Hive’s ACID tables.Apache ParquetOpen sourceThe standard columnar file format for analytics: each column stored and compressed on its own, with statistics that let engines skip data.Apache TsFileOpen sourceA columnar file format for device time series, with per-series indexes and time-series encodings; the storage format of Apache IoTDB and its own Apache project since 2023.PuffinOpen sourceApache Iceberg’s file format for the data that doesn’t fit in a manifest: a container of typed binary blobs, such as distinct-value sketches and deletion vectors, described by a JSON footer.

How it fits with other technologies

Convert Parquet to Vortex and fetch three rows · python

import pyarrow.parquet as pq
import vortex as vx   # pip install vortex-data

vx.io.write(pq.read_table("trips.parquet"), "trips.vortex")

vf = vx.open("trips.vortex")
everything = vf.scan().read_all()

# random access: read only these rows (ordered, unique)
rows = vf.scan(indices=vx.array([1, 2, 10])).read_all()

lf = vf.to_polars()   # a Polars LazyFrame with pushdown (experimental)

Vortex or Parquet?#

ParquetVortex
CompressionEncodings plus general-purpose codecs (Snappy, Zstd and others) on each pageLightweight encodings (FastLanes, FSST, ALP, dictionary, run-length), cascaded and chosen by sampling
Reading one rowDecompress and decode the page that holds itJump to the position inside the encoded data
FilteringSkip row groups and pages by statistics, then decompress the restSkip by zone maps, then evaluate many filters on the compressed data
MetadataA Thrift footer (FileMetaData) with offsets for every column chunkA FlatBuffer footer that can be read lazily, column by column, for very wide tables
Extending itNew encodings need every reader to changeEncodings and layouts are plug-ins, versioned through editions
MaturityRead by nearly every engine and warehouseFormat stable since May 2025; library still 0.x; Iceberg support coming

Read and write Vortex from DuckDB#

DuckDB’s vortex extension (DuckDB 1.4.2 and later, Linux and macOS) reads and writes the files with SQL.

sql
INSTALL vortex;
LOAD vortex;

COPY (FROM 'trips.parquet') TO 'trips.vortex' (FORMAT vortex);

SELECT passenger_count, avg(fare_amount)
FROM read_vortex('trips.vortex')
GROUP BY passenger_count;

See it in a real stack

No architecture uses Vortex yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2024–2026

SpiralDB built Vortex while designing Spiral, an object-store native data system for AI workloads. The founders’ question was where storage ends and compute begins. Modern GPUs can take in terabits per second, but formats like Parquet need a CPU in the middle to decompress everything first. They wanted a format that stays compressed as long as possible, through memory, disk and the network, and that could one day be decoded straight from object storage onto a GPU.

The repository started in February 2024. Rather than invent new compression, the team built on published database research such as FastLanes, FSST, ALP and BtrBlocks, and wrote about several of them on Spiral’s blog as they went. One of the first things they built was a logical type system kept separate from the physical encoding, unlike Arrow, where the type also fixes the memory layout. In April 2025 they described the file format planned for 1.0, and release 0.36.0 in May 2025 froze its first edition.

In August 2025 SpiralDB donated Vortex to the Linux Foundation as an LF AI & Data incubation project, backed by Microsoft, Snowflake, Palantir and NVIDIA; Microsoft reported 30% shorter Spark runtimes using Vortex inside Apache Iceberg. Arrow co-creator Wes McKinney sits on its technical steering committee. The project now lives in its own vortex-data GitHub organization, under a Linux Foundation technical charter.

Why it’s built this way

Separate logical types from physical encodings
Arrow ties each type to one memory layout. Vortex needed a column of strings to be held as FSST-compressed buffers, a dictionary or plain Arrow, while still meaning “a column of strings” to every function.
Compute on compressed data
Pruning by statistics only skips data. Vortex has a registry of compute kernels that encodings can override, so a comparison can run on dictionary codes or on ALP’s integer form instead of decompressing first; Spiral reported an 80% speedup for float comparisons this way.
Layouts are separate from the file
A layout only says which segments exist, not where they live, so the same structure can sit in a Vortex file, in a cache, or even in Postgres block storage, which ParadeDB explored.
Editions instead of a frozen feature list
To keep adding encodings without breaking readers, writers target a dated edition. Readers can always read older editions, and a writer can be held back to an edition that older readers understand.

Timeline

  1. 2024

    The Vortex repository starts at SpiralDB (February)

  2. 2025

    “Towards Vortex 1.0” describes the file format (April)

  3. 2025

    Release 0.36.0 freezes the first edition; files stay readable from here on (May)

  4. 2025

    Donated to the Linux Foundation as an LF AI & Data incubation project (August)

  5. 2026

    Release 0.87.0, with DuckDB, DataFusion, Spark and Trino integrations (October)

Sources16 · facts checked 6 October 2026

Improve this page on GitHub