cloudparts

Deep Lake

Activeloop’s storage format and database for AI data: images, video, audio, text and embeddings stored as compressed columns on object storage, searchable by vector and streamable straight into PyTorch or TensorFlow.

What it is
Vector databases
Used for
Data lakes,Retrieval-augmented generation (RAG)
Type
Open source
Made by
Activeloop
License
Apache 2.0 since version 4.0 (October 2024); Mozilla Public License 2.0 for 3.x and earlier. The hosted Activeloop service is commercial
Facts checked
24 October 2026
Website Source code

What it is

Deep Lake stores a dataset as columns of tensors, chunked and compressed, on S3, Google Cloud Storage, Azure, any S3-compatible store such as MinIO, local disk or Activeloop’s own cloud. Images, audio and video can stay in their native compression (JPEG, MP3, MP4) inside the chunks, next to labels, annotations, text and embeddings. The library is a client: it reads only the chunks a query or training loop needs, so there is no server to run for the open-source version. It versions data like Git, with commits and branches, and its dataloaders stream batches over the network into PyTorch and TensorFlow while a model trains. A query language, TQL, filters, joins and runs vector similarity search over a dataset, and LangChain and LlamaIndex use Deep Lake as a vector store for retrieval-augmented generation.

Version 4.0 (October 2024) moved the core to C++, added eventual consistency for concurrent writers, broke API compatibility with 3.x and switched the licence to Apache 2.0. The repository now also contains pg_deeplake, a PostgreSQL extension that adds a deeplake table access method and index for vector, BM25 full-text and hybrid search. Activeloop markets the hosted product as a GPU-native database for AI agents, alongside Postgres and DuckDB; packages on PyPI kept coming through August 2026 (4.7.4), while the last commit to the public GitHub repository was in May 2026.

Why teams pick it

  • Keeps raw media and its embeddings, labels and metadata in one dataset, instead of a vector database plus a separate file store
  • Serverless by design: the open-source library works directly against your own bucket, with nothing to deploy
  • Streams training data to PyTorch and TensorFlow and versions datasets with commits and branches
  • Ready integrations as a vector store in LangChain and LlamaIndex

Watch out for

  • Two incompatible generations: 4.x changed the API and format, so 3.x code and datasets need converting
  • All computation runs in the client, so query speed depends on the client’s connection to object storage
  • Activeloop’s focus has moved to its hosted GPU database, and the open repository has fallen behind the PyPI releases
  • A smaller ecosystem than general vector databases or table formats like Lance, Iceberg or Delta Lake

Good for

Specs

Compare with alternatives →
Data model
Columns of tensors: media, labels, text and embeddings
Query language
TQL: filters, joins and vector similarity
Consistency
Eventual consistency for concurrent writers (since 4.0)
Storage format
Chunked, compressed tensor columns on object storage or local disk
Written in
C++ core (since 4.0)
Ways to run it
A client library over your own bucket, or Activeloop’s hosted service

Alternatives

What all vector databases have in common →Compare side by side →
LanceOpen sourceAlso good for similarity search over embeddingsA file and table format for AI data: columnar like Parquet, but fast at fetching single rows, with vector and full-text indexes, versions and room for images and video.ChromaOpen sourceAlso good for similarity search over embeddingsAn open-source vector database for AI apps that runs inside a Python or JavaScript process, as a single server, or as a distributed service on object storage.PineconeCommercialAlso good for similarity search over embeddingsA managed, serverless vector database that keeps its indexes in object storage and serves dense-vector, sparse-vector and BM25 full-text search from the same index.turbopufferCommercialAlso good for similarity search over embeddingsA managed vector and full-text search engine that keeps every index in object storage and caches the active parts on NVMe and in memory, trading occasional slow cold queries for a much lower price per GB.VectorizeCommercialAlso good for similarity search over embeddingsCloudflare’s serverless vector database for Workers, which stores embeddings in R2-backed indexes and bills only for vector dimensions stored and queried.

How it fits with other technologies

pg_deeplake: a Deep Lake-backed table in PostgreSQL with a vector index · sql

CREATE EXTENSION pg_deeplake;

CREATE TABLE vectors (
    id SERIAL PRIMARY KEY,
    v1 float4[],
    v2 float4[]
) USING deeplake;

CREATE INDEX index_for_v1 ON vectors USING deeplake_index (v1 DESC);

INSERT INTO vectors (v1, v2) VALUES
    (ARRAY[1.0, 2.0, 3.0], ARRAY[1.0, 2.0, 3.0]),
    (ARRAY[4.0, 5.0, 6.0], ARRAY[4.0, 5.0, 6.0]);

-- <#> is cosine similarity
SELECT id, v1 <#> ARRAY[1.0, 2.0, 3.0] AS score
FROM vectors
ORDER BY score DESC
LIMIT 10;

See it in a real stack

No architecture uses Deep Lake yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2019–2026

Training deep learning models means moving millions of images, videos and audio clips from cloud storage to GPUs fast enough to keep them busy. Data lakes built for tables, with Parquet files and SQL engines, were not designed for tensors and raw media. Activeloop published its first open-source library for this, hub, on PyPI in August 2019: datasets stored as chunked arrays in object storage that train code could index like NumPy arrays and stream from the cloud.

In 2022 the project took the name Deep Lake, and a paper by Davit Buniatyan and the Activeloop team described it as a lakehouse for deep learning, with time travel, a Tensor Query Language, in-browser visualization and streaming to PyTorch, TensorFlow and JAX. LangChain and LlamaIndex later added it as a vector store for LLM apps. Version 4.0 in October 2024 rewrote the core in C++ and moved to Apache 2.0; by 2026 Activeloop was selling a hosted, GPU-native database for AI agents built on the format.

Why it’s built this way

Store data as compressed tensor chunks
Training reads samples in batches, and media is already compressed; chunking arrays and keeping native compression lets a dataloader fetch exactly the bytes a batch needs without decoding whole files.
Compute in the client, not a server
Data already sits in the team’s own bucket, so a library that reads it directly avoids running and paying for a database server, and works the same locally and in any cloud.
Version datasets like code
Models are only reproducible if you know which version of the data they saw, so datasets get commits, branches and diffs.

Timeline

  1. 2019

    Activeloop publishes the first hub package on PyPI in August

  2. 2022

    Renamed Deep Lake; paper describes it as a lakehouse for deep learning; version 3.0

  3. 2024

    Deep Lake 4.0 in October: C++ core, new API, Apache 2.0

  4. 2026

    Repository gains the pg_deeplake PostgreSQL extension; 4.7.4 on PyPI in August

Sources8 · facts checked 24 October 2026

Improve this page on GitHub