Cloud Parts

Lance

A file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.

What it is
Table formats,File formats
Used for
Lakehouses
Type
Open source
Made by
Lance community, started by LanceDB
License
Apache 2.0
Facts checked
5 October 2026
Website Source code

What it is

Parquet is built for scans: read a few columns of every row. AI workloads also need the opposite. Training samples random rows, search fetches the 10 nearest matches, and a pipeline backfills a new embedding column. Lance is a columnar file format that keeps scans fast but drops Parquet’s row groups, using encodings that let a reader jump straight to one row. Its authors measured this in a 2025 paper, and the project claims around 100× faster random access than Parquet.

On top of the files sits a table format, much like Iceberg or Delta Lake. Rows are grouped into fragments, and each fragment can have several data files that each hold some of its columns, so adding a column means writing new files rather than rewriting old ones. Every commit writes a new manifest in _versions/, which gives ACID transactions, time travel, tags and branches. Deletes are recorded in deletion files instead of rewriting data. Indexes are part of the spec and live in _indices/: vector indexes such as IVF_PQ and HNSW, full-text search, and scalar indexes such as B-tree and bitmap. Images, audio and video can sit in the same table as blobs, next to their embeddings and labels.

Why teams pick it

  • Random reads of single rows are fast, so shuffled training and search don’t need a separate copy of the data
  • Vector, full-text and scalar indexes are stored with the table, so search runs on object storage without a separate vector database
  • Adding or backfilling a column writes only that column’s new files
  • Versions, tags and branches give reproducible training sets and easy rollbacks
  • Images, video, audio and embeddings live alongside ordinary columns

Watch out for

  • Far fewer engines read it than Parquet or Iceberg
  • Compacting files to remove deleted rows moves rows to new addresses, so every index has to be remapped, and compaction can clash with index updates
  • Most tooling is Python and Rust; the Spark, Trino and Flink connectors are younger
  • Built for AI and search data; for plain BI tables, Parquet under Iceberg or Delta Lake is the safer default

Alternatives

What all table formats have in common →
Apache HudiOpen sourceA table format built for upserts and incremental pulls, with record-level indexes and built-in table services.Apache IcebergOpen sourceThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.Apache PaimonOpen sourceA lake table format built on LSM trees, so streaming jobs can upsert into object storage and other jobs can read the changes as a stream.Apache XTableIncubatingOpen sourceAn incubating Apache project, formerly Onehouse’s OneTable, that makes one set of Parquet files readable as a Delta Lake, Apache Iceberg and Apache Hudi table at once by translating only the table metadata.Delta LakeOpen sourceAn open table format built around a transaction log that lives in the table’s own folder; the default on Databricks.DuckLakeOpen sourceA lakehouse format that keeps table metadata in an ordinary SQL database and the data in Parquet files.icepickExperimentalOpen sourceA small, experimental Rust CLI and library that commits Parquet files to Apache Iceberg tables and runs compaction and snapshot cleanup through a REST catalog, aimed at Cloudflare’s R2 Data Catalog (now Basin Catalog) and Amazon S3 Tables, with no JVM.

How it fits with other technologies

Write a Lance table, search it and read an old version · python

import lance

# from a pandas DataFrame, Arrow table or Parquet dataset
ds = lance.write_dataset(images, "s3://acme-ml/images.lance")

# a vector index, stored inside the table
ds.create_index("embedding", index_type="IVF_PQ", num_partitions=256, num_sub_vectors=16)

# the 10 rows closest to a query vector
hits = ds.to_table(columns=["id", "label"], nearest={"column": "embedding", "q": query, "k": 10})

# every write makes a new version; tag one and read it later
ds.tags.create("train-v1", ds.version)
old = lance.dataset("s3://acme-ml/images.lance", version="train-v1")

Lance or Parquet in a table format?#

Parquet + Iceberg or Delta LakeLance
Best atScanning a few columns of many rowsScans and fetching individual rows by position
Inside a fileRow groups, each with a chunk per column and min/max statisticsLarge pages per column with no row groups; statistics live in indexes instead
Add a column with dataRewrite the files, or add it emptyWrite new files holding just that column
SearchFilters and partition pruningVector, full-text and scalar indexes stored in the table
Images and videoUsually links to files elsewhereStored in the table as blobs, read lazily
VersionsSnapshots and time travelVersions, tags and branches
Who reads itAlmost every engine and warehousePython (pandas, Polars, PyTorch, Ray), DuckDB, DataFusion, Spark, Trino, Flink

Query and search a Lance table from DuckDB#

DuckDB’s lance extension reads, writes and searches Lance tables with SQL.

sql
INSTALL lance;
LOAD lance;

COPY (FROM 'images.parquet') TO 's3://acme-ml/images.lance' (FORMAT lance);

SELECT label, count(*) FROM 's3://acme-ml/images.lance' GROUP BY label;

-- the 5 nearest rows to a query vector
SELECT id, label, _distance
FROM lance_vector_search('s3://acme-ml/images.lance', 'embedding', $query, k = 5)
ORDER BY _distance;

See it in a real stack

No architecture uses Lance yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2021–2026

Around 2021, Chang She and Lei Xu asked why working with embeddings, images and video was still so much harder than working with tables. The standard formats were built for something else. Parquet is designed for scanning columns: its encodings aren’t “sliceable”, so reading one row usually means loading a whole page, which hurts shuffled training, search results and other point lookups. And when one column holds 4 KiB embeddings or whole images, no row group size works well. They decided to start from first principles, setting aside formats like Parquet and WebDataset, and the idea became LanceDB and its open format, Lance.

The first version paired a columnar file with a versioned table. In April 2024 the team announced Lance v2, a new file format, because v1 “still struggles in a number of cases”. In April 2025 a paper by Weston Pace, Chang She, Lei Xu and others measured its approach to random access against Parquet and Arrow.

As AI labs and companies such as Runway, Midjourney, Netflix, Uber and ByteDance adopted it, LanceDB set up formal community governance in November 2025, with a project management committee modelled on the Apache Way. The format moved to its own website (lance.org) and GitHub organization (lance-format), separate from LanceDB’s database.

Why it’s built this way

No row groups; fast single-row reads
Row groups force a size trade-off that wide values break. Lance stores large pages per column and uses encodings a reader can jump into, so fetching one row doesn’t mean decoding a whole page.
Encodings are extensions, not part of the reader
Adding an encoding to Parquet means changing every reader across many implementations. In Lance v2 encodings are plug-ins, so new ones can arrive without a new file format.
Add columns by writing new files
ML pipelines often backfill a new embedding or label column. Each fragment can hold several data files with different columns, so adding a column attaches new files instead of rewriting the table.
Indexes are part of the table
Vector, full-text and scalar indexes live in the dataset’s _indices/ folder and are versioned with it, so search runs straight from object storage without a separate vector database.

Timeline

  1. 2021

    Chang She and Lei Xu set out to build storage for multimodal AI data

  2. 2022

    The open-source Lance repository starts in July

  3. 2024

    Lance v2 file format announced in April

  4. 2025

    Paper on random access (April); LanceDB raises a $30M Series A in June

  5. 2025

    Community governance; the format moves to lance.org and the lance-format org

  6. 2026

    Apache Hudi 1.2 adds Lance as a file format for vector and multimodal data

Sources16 · facts checked 5 October 2026

Improve this page on GitHub