Cloud Parts

Hotdata

A managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.

What it is
Query engines
Type
Commercial product
Made by
Hotdata
License
Proprietary managed service; the CLI is MIT, and the DuckLake library it donated to DataFusion is Apache 2.0
Facts checked
11 October 2026
Website

What it is

Hotdata is a hosted query engine reached at api.hotdata.dev. An organization holds workspaces, and each workspace is an isolated execution environment that shares no compute, cache or credentials with any other. Inside it you create databases, lightweight query scopes made in one call that can delete themselves after an expires_at such as 24 hours; the intended pattern is one per job or per agent session. Data gets in two ways. You push files (CSV, JSON or Parquet) through an upload endpoint, where each publish atomically replaces, appends, upserts, updates or deletes rows by key. Or you set up ingests that pull from 168 source types in eight families: SQL databases and warehouses (Postgres, MySQL, SQL Server, Oracle, Snowflake, BigQuery, Databricks, Redshift), REST APIs, files in S3, GCS or Azure, Iceberg catalogs (REST, Glue, Polaris, Unity Catalog), Delta Lake, DuckLake, Kafka topics including Debezium change logs, and tables derived from other tables. A database can attach another database’s catalog read-only, so one query joins both, and since July 2026 it can be forked into an independent copy. Tables are stored with DuckLake through datafusion-ducklake, a library Hotdata wrote and donated to the DataFusion contrib project in June 2026.

SQL is read-only: INSERT, COPY and DDL are refused, and writes go through loads. The native dialect, HotSQL, is Postgres-like and built on DataFusion, and a query can instead be written in PostgreSQL, DuckDB or Snowflake syntax and translated before it runs. It adds BM25 full-text search, PostGIS-style ST_ functions and vector search. Vector search runs on one node: an HNSW graph in USearch, scalar columns in SQLite and the full rows in Parquet, all sharing row keys. With a WHERE clause it first scans Parquet for matching keys, then either walks the graph with those keys as a filter (more than 5% of rows match) or computes exact distances for each key (5% or fewer). Long queries run asynchronously, and every result is saved and can be fetched again as Arrow, Parquet, CSV, Markdown or JSON.

The company was founded by Eddie A Tejeda (CTO), Divya Ranganathan (CEO) and Zac Farrell, with Andrew Lamb, PMC chair of Apache DataFusion, as an advisor. Its site went live in February 2026, the CLI in March and managed databases in May. Pricing is usage-based: a $5 monthly base fee, storage ($0.20 a month for 10 GB) and bytes scanned ($10.24 per TB), with a $100 credit at sign-up. The engine is closed source; the CLI, SDKs and integrations for Ibis, LangChain, dlt, dbt and Grafana are on GitHub.

Why teams pick it

  • Isolation by design: workspaces share nothing, databases are cheap to create, fork and expire, and attached catalogs are read-only, so many agents can work side by side without touching the sources
  • One SQL surface for analytics, BM25 full-text, filtered vector search and geospatial queries, and it accepts PostgreSQL, DuckDB and Snowflake dialects as well as its own
  • Pulls from 168 source types, from Postgres and Snowflake to Iceberg catalogs and Kafka, and saves every result so later steps can reuse it as data
  • In its own TPC-H benchmark (19 of 22 queries, July 2026), total execution time stayed between 2.59 and 2.90 s from 1 to 50 concurrent streams, while Snowflake rose from 2.88 to 9.26 s

Watch out for

  • Very young: the site launched in February 2026 and managed databases in May 2026, and the closed-source engine is available only as Hotdata’s service
  • No write SQL at all: data arrives only through loads and ingests, the first load into a table must replace it, row-at-a-time writes are discouraged, and whether append and keyed loads work depends on how the instance is provisioned
  • Built for small, fast data, not warehouse scans: its founders target datasets under about 100 GB, and vector search runs on a single node sized for 1 to 10 million vectors
  • Its benchmark numbers are vendor-run: TPC-H Q18 failed on Hotdata at every concurrency level, and Q20 and Q21 were left out of the totals because more than about half their runs failed under concurrency

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.RayforceOpen sourceA small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

What Hotdata uses

Load a Parquet file into an instant database, query it, search it, then delete it (from the Quick Start; not run, it needs a Hotdata account) · bash

brew install hotdata-dev/tap/cli
hotdata auth login                      # opens a browser to sign in

# An instant database with one declared table, loaded from a URL
hotdata databases create --name airbnb --catalog airbnb --table listings
hotdata databases load --catalog airbnb --table listings \
  --url https://hotdata.dev/data/sf-airbnb-listings.parquet

# HotSQL; tables are addressed as <catalog>.<schema>.<table>
hotdata query "SELECT count(id) AS total_rows FROM airbnb.public.listings"

# A BM25 full-text index on one column, then a search against it
hotdata search create listings_description --type text \
  --from airbnb.public.listings --column description
hotdata search "cozy apartment with a view" \
  --index listings_description --select id,name --limit 10

# Throw the database away when the job is done
hotdata databases remove airbnb

See it in a real stack

No architecture uses Hotdata yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2026

Hotdata comes from a bet that AI agents use data differently from people. In “Why we are building Hotdata” (April 2026), Eddie A Tejeda argued that a decade of infrastructure had optimized for throughput and scale, while vectorized execution, columnar formats, SIMD and fast NVMe let one machine process tens of millions of rows in under 10 ms. Agents, Tejeda wrote, create many datasets, issue large numbers of small queries and chain the results, so latency, context and composability matter more than raw throughput. A follow-up post added that one slow query can stall dozens of dependent steps in an agent’s plan.

The team built the engine on Apache DataFusion, a Rust query engine library, because it leaves storage, catalogs, indexing and caching to the builder. The engine’s caching layer landed in January 2026, the public site and geospatial SQL in February, the CLI with BM25 and vector search in March, and managed databases with automatic expiry in May. For table metadata it chose DuckLake over Iceberg: in June 2026 it released datafusion-ducklake to the DataFusion contrib organization and made it Hotdata’s storage backend. In July it added database forks and mutable tables and published a concurrency benchmark against Snowflake and BigQuery.

Why it’s built this way

Build on DataFusion instead of a finished database
Tejeda wrote that DuckDB is a complete database, while DataFusion provides planning, optimization and execution but lets the builder decide how data is stored, indexed, cached and planned, which is the control Hotdata wanted.
DuckLake instead of Iceberg for table metadata
Reading an Iceberg table means walking metadata files, manifest lists and manifests in object storage before touching data. DuckLake keeps that metadata in a SQL database, so one query finds the Parquet files, which suits small, fast queries.
Vector search as a SQL operator, not a separate service
Inspired by pgvector, the team wanted WHERE filters and ORDER BY l2_distance(…) LIMIT k in one query, aiming for under 100 ms with filters. It keeps three stores keyed alike (USearch, SQLite, Parquet) and picks a strategy from the filter’s selectivity.
Short-lived, isolated databases with durable results
Agents lose their place when intermediate results disappear or change between runs, so every result is saved and replayable, and each job gets its own database that can expire on its own.

Timeline

  1. 2026

    January: the caching layer lands, and query results become queryable data

  2. 2026

    February: hotdata.dev goes live, with geospatial SQL

  3. 2026

    March: the CLI launches, with BM25 full-text and vector search

  4. 2026

    May: managed instant databases launch

  5. 2026

    June: datafusion-ducklake is donated to DataFusion contrib and becomes the storage backend

  6. 2026

    July: database forks, mutable tables and a concurrency benchmark

Sources17 · facts checked 11 October 2026

Improve this page on GitHub