Cloud Parts

Rayforce

A small embeddable engine in pure C that runs columnar analytics, graph traversals and recursive Datalog queries in one optimised plan.

What it is
Query engines,Graph databases
Type
Open source
Made by
Anton Kundenko, with Lynx Trading Technologies
License
MIT
Facts checked
6 October 2026
Website Source code

What it is

Rayforce is a C library, plus a REPL binary, that you link into a program. You describe a query as a lazy graph of operations, either through the C API, through Rayfall (its own Lisp-like query language) or through Python bindings: scans, filters, joins, group-bys, window functions and graph steps such as expanding neighbours or finding shortest paths. Nothing runs until you execute it. A multi-pass optimiser first rewrites the whole plan, pushing filters and column choices down and passing selections sideways between joins and graph expansions. Element-wise parts are then compiled to bytecode and run over 1,024-row chunks that stay in the CPU’s L1 cache, spread across a thread pool. Graph edges are stored next to the tables as compressed sparse rows (CSR), indexed both forwards and backwards, so one query can aggregate columns and then walk relationships without handing data to a second system. Everything goes through its own memory allocator instead of malloc, and there are no external dependencies.

Tables live in memory or on disk as one file per column (splayed tables) or split into date partitions, the layouts kdb+ uses, memory-mapped and backed by a write-ahead log. It reads CSV in parallel and imports Parquet, has an IPC client and server built in, and an add-on speaks q’s IPC wire format. Rayforce is MIT licensed, written mainly by Anton Kundenko and co-developed with and sponsored by Lynx Trading Technologies. There’s no paid edition; a managed Rayforce Cloud is announced with no launch date. The current release is v2.11.0, from October 2026.

Why teams pick it

  • Tiny and self-contained: pure C, one public header, zero dependencies, its own allocator
  • Relational operators, graph algorithms (BFS, Dijkstra, PageRank, Louvain, worst-case-optimal joins), recursive Datalog and HNSW vector search share one optimiser
  • kdb+-style on-disk tables: column files, date partitions, memory mapping and a transaction log
  • MIT licensed, with bindings for Python, Rust and WebAssembly

Watch out for

  • No SQL: queries are written in Rayfall, the C API or Python, so BI tools and SQL clients can’t connect
  • Young and fast-moving: the reworked v2 line was first tagged in June 2026 and has had ten more minor releases since, with one author writing most commits
  • Parquet support covers local files only; there’s no reading straight from object storage
  • Prebuilt binaries only for Linux x86-64 and macOS arm64; elsewhere you build from source on a POSIX system
  • A small community, and the production users it names are listed by the project itself

Alternatives

What all query engines have in common →
Amazon AthenaAWSAWS’s serverless SQL engine for data in S3: no cluster to run, and on-demand queries cost $5 per TB scanned.Apache Arrow AceroExperimentalOpen sourceThe streaming execution engine inside Arrow’s C++ library: you hand it an exact plan of nodes (scan, filter, project, aggregate, join, sort) and it streams Arrow record batches through them, with no SQL parser or optimizer of its own.Apache CalciteOpen sourceA Java framework with everything a database needs to understand a query (SQL parser, validator, relational algebra and a rule-based, cost-based optimizer) but no storage of its own, which Hive, Flink, Druid, Dremio, Beam and many other systems embed as their SQL front end and planner.Apache DataFusionOpen sourceA query engine library in Rust, built on Arrow, that other projects embed and extend to build their own databases and data tools.+1 Apache DataFusion CometApache HadoopOpen sourceThe project that made big-data clusters normal: a distributed file system (HDFS), a cluster scheduler (YARN) and the MapReduce engine.Apache HiveOpen sourceThe first SQL engine for Hadoop, and the source of the Hive Metastore, the table catalog that Trino, Spark, Impala and Flink still read.Apache ImpalaOpen sourceAn MPP SQL engine in C++ for interactive queries over Hive tables on HDFS, Kudu, Iceberg and object storage; built and mainly sold by Cloudera.Apache PigDormantOpen sourceA step-by-step scripting language, Pig Latin, that compiles to MapReduce, Tez or Spark jobs on Hadoop. Still released, but the project calls itself dormant.Apache SparkOpen sourceThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.Apache ZeppelinOpen sourceA self-hosted web notebook where each paragraph runs through an interpreter: Spark, Flink, JDBC, Python and more. Maintained, but behind Jupyter and Databricks in reach.chDBOpen sourceThe ClickHouse engine packaged as a library: run ClickHouse SQL inside a Python, Go, Rust or Node.js program, with no server.Cloudflare Basin SQLRenamedCommercialCloudflare’s serverless, distributed SQL engine for Apache Iceberg tables in Basin Catalog on R2, called R2 SQL until it became generally available on 1 October 2026, billed by the data it scans.DuckDBOpen sourceAn in-process analytical SQL database: a library with no server that queries Parquet, CSV, Iceberg and S3 directly.GizmoSQLOpen sourceA database server that wraps DuckDB (or SQLite) and serves it to many clients over Arrow Flight SQL, with TLS and authentication.GlareDBDormantOpen sourceA small, embeddable SQL engine in Rust for querying Parquet and CSV files locally, over HTTP or in S3 and GCS, from a CLI, Python or the browser. It began as a DataFusion-based federated query service and was rewritten in 2025 on its own engine, and it has had no release since June 2025.HotdataCommercialA managed SQL engine for AI agents, built in Rust on Apache DataFusion and Arrow: each agent gets short-lived “instant databases” to load data into, then runs read-only SQL, vector, full-text and geospatial queries over an HTTP API or CLI.IbisOpen sourceA Python dataframe API that compiles to SQL, so the same code runs on DuckDB on a laptop and on BigQuery, Snowflake, PostgreSQL or more than 20 other engines.PolarsOpen sourceA DataFrame library with a Rust query engine underneath: you write Python (or Rust, SQL, Node.js, R) and it plans, optimises and runs the query on every core, streaming data bigger than memory.Presto (PrestoDB)Open sourceFacebook’s distributed SQL engine, now governed by the Linux Foundation, moving its workers from Java to C++ on Velox. Trino is its 2019 fork.Spice.ai OSSOpen sourceA single-binary SQL engine in Rust, built on DataFusion and Arrow, that federates queries across databases, warehouses and lakes, keeps hot datasets in a local accelerated copy, and adds search and LLM APIs on top.TrinoOpen sourceA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.VeloxOpen sourceMeta’s open-source C++ execution engine library: it takes an already-optimized query plan and runs it with vectorized operators, and it is the engine inside Presto’s C++ workers (Prestissimo) and Apache Gluten’s native Spark.

How it fits with other technologies

A group-by and a recursive graph query in one Rayfall script · bash

brew install rayforcedb/tap/rayforce

cat > flows.rfl <<'EOF'
; columnar: total quantity per symbol
(set t (table [Symbol Side Qty]
  (list [AAPL GOOG MSFT AAPL GOOG] [Buy Sell Buy Sell Buy] [100 200 150 300 250])))
(show (select {from: t by: Symbol Qty: (sum Qty)}))

; graph: every account that money from account 1 can reach
(set db (datoms))
(set db (assert-fact db 1 'pays 2))
(set db (assert-fact db 2 'pays 3))
(set db (assert-fact db 3 'pays 4))
(rule (reaches ?a ?b) (?a :pays ?b))
(rule (reaches ?a ?c) (?a :pays ?b) (reaches ?b ?c))
(show (query db (find ?b) (where (reaches 1 ?b))))
EOF

rayforce flows.rfl

One engine, four kinds of query#

Each of these builds into the same lazy plan, so the optimiser sees the whole workload at once.

Columnar analytics

select, joins, windows

Vectorised filters, radix-partitioned hash joins, group-bys, window functions, pivots and as-of joins over typed column vectors.

Graph algorithms

CSR storage

BFS, DFS, Dijkstra, A*, PageRank, Louvain communities, betweenness and Leapfrog TrieJoin over edges stored as forward and reverse CSR arrays.

Datalog

recursive rules

Facts stored as entity, attribute, value triples; recursive rules give reachability and transitive closure, evaluated to a fixpoint.

Vector search

HNSW

An HNSW index with cosine, L2 and inner-product metrics; a where filter is pushed into the search so rejected candidates don’t use up result slots.

See it in a real stack

No architecture uses Rayforce yet. Drop it into the playground and wire it up yourself.Open playground →

Benchmarks

Benchmark · ClickHouse Inc.ClickBenchHow fast analytical databases run 43 SQL queries over one 100-million-row table of web analytics events, on the same cloud machines.Includes Rayforce · 7 caveats to read first
How to read a benchmark →
Where it came from2023–2026

Analytics stacks often split work between a columnar engine for tables and a separate library or service for graphs, each with its own data model, language and memory layout. Joining them means serialising data between processes, and no optimiser ever sees the whole pipeline. Rayforce was built on the bet that one engine, with one optimiser, one execution model and one allocator, could do both faster than the seam between two.

Anton Kundenko started Rayforce in February 2023 as a columnar database in pure C, with a Lisp-like language instead of SQL, aimed at financial and time-series data. Its vocabulary of splayed and partitioned tables and a q-style transaction log echoes kdb+ and its q language. Lynx, a trading technology firm, co-develops and sponsors it. Python, Rust and WebAssembly bindings followed in 2025. In April 2026 a large rework, the v2 line, rebuilt the aggregation engine and storage; the project now describes itself as columnar analytics and graph traversal in one pipeline. Its first tagged release, v2.1.0, came in June 2026, and the same month an add-on let it talk to q clients and servers. PeachQ, an open implementation of the q language, runs on it.

Why it’s built this way

Pure C with no dependencies and its own allocator
A buddy allocator with slab caches and per-thread heaps avoids malloc contention, and a library with no third-party code can be linked into anything with a C FFI.
Graph traversals as operators in the same plan as joins
Moving data between a table engine and a graph library costs serialisation on every step and hides the full query from the optimiser.
A small Lisp-like language instead of SQL
Rayfall keeps the syntax small, and its select, update and pivot forms map directly onto the engine’s operation graph.

Timeline

  1. 2023

    Anton Kundenko starts Rayforce in February; MIT licensed from March

  2. 2025

    Rust, Python and WebAssembly bindings

  3. 2026

    April: the v2 rework; June: first tagged release, v2.1.0

  4. 2026

    June: an add-on speaks q’s IPC wire format

  5. 2026

    October: v2.11.0

Sources18 · facts checked 6 October 2026

Improve this page on GitHub