cloudparts

NewSQL and distributed SQL

Relational databases that spread tables across many machines and keep SQL and ACID transactions, so writes scale out without sharding by hand.

2 technologies on this page ↓

How it works

Underneath the SQL, most of these systems turn every row and index entry into a key-value pair in one sorted keyspace, and cut that keyspace into ranges of contiguous keys: ranges in CockroachDB (up to 512 MiB), Regions in TiDB (256 MiB by default), tablets in YugabyteDB, splits in Spanner. Each range is copied to three or more nodes and forms its own consensus group (Raft, or Paxos in Spanner): one replica leads, and a write commits once a majority of replicas have it, so losing a node or a zone needs no manual failover. Ranges split when they grow and move when a node is busy, which is how adding machines adds write capacity. The SQL layer is usually stateless: any node accepts a connection, plans the query and sends the pieces to the ranges that hold the data. A transaction that touches several ranges commits with a two-phase protocol, and every version of every row carries a timestamp (MVCC), so reads see a consistent snapshot without blocking writers. The hard part is agreeing on time across machines. Spanner uses TrueTime, clocks backed by GPS and atomic clocks with a known error bound, and waits out that bound before acknowledging a commit. CockroachDB and YugabyteDB use hybrid logical clocks over ordinary NTP; CockroachDB restarts a read that meets a value inside the clock-uncertainty window. TiDB asks one service, the Placement Driver, for every timestamp. FoundationDB, a transactional key-value store rather than a SQL database, has a single sequencer hand out commit versions; SQL comes from layers on top, such as the Record Layer’s JDBC interface. Aurora DSQL builds hybrid logical clocks on the microsecond-accurate Amazon Time Sync Service. The price of all this is latency and retries: every write waits for a quorum, possibly in another region, conflicting transactions fail with a serialization error (SQLSTATE 40001) that the application must retry, and an ever-increasing key, such as a timestamp or sequence, sends every insert to the same range.

Think of it as

A library that has outgrown its building and moved into branches across the city. Each shelf of books is copied to three branches, and a change to the catalogue counts once two of them have written it down, so one branch burning down loses nothing. You can ask at any branch and still check out books from several shelves at once, but a loan that spans branches takes a few phone calls, and everyone has to agree on what time it is.

Reach for it when

  • One primary can no longer take the write load, and splitting the data by hand would put joins and cross-shard transactions into application code
  • The database must survive losing a node, a zone or a whole region with no data loss and no manual failover
  • Users or regulations are spread across regions, and rows should live near their users while staying in one logical database
  • You need strict correctness (serializable or stricter transactions) at a scale where NoSQL stores would make you handle inconsistency yourself

Think twice when

  • The data fits on one well-sized PostgreSQL or MySQL server with replicas: that is simpler, faster per query and fully compatible
  • Latency-critical writes from one place: a quorum write across zones or regions costs more round trips than a commit on a local disk
  • Analytics over large scans; use a warehouse or an HTAP column store such as TiDB’s TiFlash, not the row store
  • Applications that depend on extensions, stored procedures or exact PostgreSQL or MySQL behaviour, which these systems implement only partly

A real example

A payments company outgrows a single PostgreSQL primary and must keep running if an AWS region fails. It moves to a distributed SQL database with nodes in three US regions and declares the database able to survive a region failure. Account rows are homed in the region where each customer signed up, so most reads and writes stay local, while a small currencies table is replicated for fast reads everywhere. A transfer between two customers in different regions now commits in tens of milliseconds instead of two, because the transaction record must reach a quorum across regions. The team switches to random UUID keys after finding that time-ordered IDs sent every insert to one range, and wraps transfers in a retry loop for serialization errors. When us-east goes dark during a drill, the other two regions keep accepting payments.

Technologies

Specific products, and what each does differently.

What each provider calls it

AWS
Aurora DSQL: serverless, PostgreSQL-compatible, active-active across Availability Zones or peered Regions, with optimistic concurrency and snapshot isolation; Aurora itself shares one storage volume across zones but keeps a single writer instance
Google Cloud
Spanner: TrueTime and Paxos, external consistency (stricter than serializable), GoogleSQL or a PostgreSQL dialect per database
Azure
No distributed SQL engine of its own; Azure Database for PostgreSQL elastic clusters run the Citus extension to shard PostgreSQL across nodes, and CockroachDB’s managed Mission Critical tier runs on Azure
Open source & others
TiDB (MySQL-compatible, Apache 2.0), YugabyteDB (PostgreSQL-compatible, Apache 2.0 core), CockroachDB (PostgreSQL wire protocol, proprietary licence since 2024), FoundationDB (Apache 2.0 transactional key-value core, with SQL through the Record Layer); Vitess and Citus shard existing MySQL and PostgreSQL servers instead

How the main systems do it#

Each splits data into replicated ranges; they differ in what they speak, how they agree on time, and the guarantee you get by default.

SpeaksReplicationOrdering transactionsDefault guaranteeLicence
SpannerGoogleSQL or PostgreSQL dialectPaxos per splitTrueTime, then wait out the clock error before acknowledgingExternal consistency, stricter than serializableGoogle Cloud service
CockroachDBPostgreSQL wire protocolRaft per range, up to 512 MiBHybrid logical clocks over NTP; reads restart inside the uncertainty windowSERIALIZABLE; READ COMMITTED optionalProprietary, source available
TiDBMySQL protocolRaft per Region (256 MiB) in TiKVCentral timestamp oracle in the Placement DriverSnapshot isolation, reported as REPEATABLE READApache 2.0
YugabyteDBPostgreSQL (YSQL reuses its query layer) and Cassandra-style YCQLRaft per tabletHybrid logical clocksSnapshot, serializable or read committedApache 2.0 core
FoundationDBKey-value API; SQL only through layersTransaction logs and storage servers; failures handled by recruiting new rolesOne sequencer hands out commit versionsStrict serializabilityApache 2.0
Aurora DSQLPostgreSQL-compatible (16)Synchronous across three Availability Zones, or peered RegionsHybrid logical clocks on Amazon Time Sync ServiceSnapshot isolation, with conflicts checked at commitAWS service

Three things people have called NewSQL#

Pavlo and Aslett’s 2016 survey sorted the systems sold under the name into three groups. Only the first is what “distributed SQL” usually means today.

New architectures

Spanner, CockroachDB, VoltDB

Databases written from scratch for a cluster of shared-nothing nodes, each managing its own storage, with their own concurrency control, replication and distributed query execution. TiDB, YugabyteDB and Aurora DSQL came later and fit here.

Transparent sharding middleware

MaxScale, ScaleArc; today Vitess

A layer that routes queries across ordinary single-node databases and presents them as one. Applications keep their driver and SQL dialect, but cross-shard transactions and joins are harder, and each node keeps its 1970s disk-oriented engine.

Database as a service on a new architecture

Amazon Aurora

Managed services that rebuilt the storage underneath a familiar engine. The survey counted Aurora, whose log-structured storage improved I/O, though it doesn’t partition writes across nodes.

In code · sql

-- Works on CockroachDB and YugabyteDB (PostgreSQL syntax); TiDB and Spanner differ in details
CREATE TABLE accounts (
  id      UUID PRIMARY KEY DEFAULT gen_random_uuid(),  -- random keys spread inserts across ranges
  owner   TEXT NOT NULL,
  balance DECIMAL(12,2) NOT NULL CHECK (balance >= 0)
);

-- One transaction across two rows that may live on different nodes:
-- both changes commit through the consensus groups of their ranges, or neither does.
BEGIN;
UPDATE accounts SET balance = balance - 25 WHERE id = '8a1c…';
UPDATE accounts SET balance = balance + 25 WHERE id = 'f03e…';
COMMIT;
-- On conflict: ERROR ... (SQLSTATE 40001). Retry the whole transaction.

Often confused with

Relational databasesSQLKey-value storesCoordination servicesWide-column storesLSM trees
Where it came from2011–2025

In the 2000s web applications outgrew single database servers. Companies such as eBay and Google wrote middleware that sharded Oracle or MySQL across many machines, but it handled single-row reads and writes well and left joins and multi-row transactions to application code. NoSQL systems such as Bigtable, Dynamo and their open-source descendants went further and dropped transactions and the relational model for eventual consistency. Some organizations, Google most notably, found that their developers then spent too much time writing code to cope with inconsistent data. Google’s answer was Spanner, described in 2012 as the first system to distribute data at global scale and still support externally consistent distributed transactions; it later grew a full SQL engine and became a public cloud service in 2017.

Matthew Aslett, an analyst at The 451 Group, coined “NewSQL” in April 2011 as shorthand for the new vendors of scalable, high-performance SQL databases, replacing the group’s earlier label ScalableSQL. He stressed that “the new thing about the NewSQL vendors is the vendor, not the SQL”, and his first list mixed clustered databases such as Clustrix, VoltDB and NimbusDB with new MySQL storage engines and hosted services. In 2016 he and Andy Pavlo defined NewSQL more narrowly, as relational databases that give NoSQL’s scalability for read-write transaction workloads while keeping ACID guarantees, and noted that several early companies had already failed or been sold.

The systems that lasted mostly followed Spanner’s design without its special hardware. CockroachDB reached 1.0 in May 2017 with hybrid clocks over NTP in place of atomic clocks, and TiDB and YugabyteDB put MySQL and PostgreSQL front ends on Raft-replicated key-value stores. YugabyteDB and CockroachDB describe themselves as distributed SQL databases, the name now more common than NewSQL. In May 2025 AWS added Aurora DSQL, a serverless PostgreSQL-compatible database that runs active-active across Regions.

Why it’s built this way

A consensus group per range of keys
Replicating a whole database through one primary limits writes to one machine and makes failover a manual or risky step. Giving every range its own Raft or Paxos group lets writes spread over many leaders, and a majority vote means a minority of failed replicas loses nothing.
Multiversion concurrency control instead of read locks
Most of these systems use MVCC, so long read-only queries see a snapshot and never block writers. The idea dates from a 1979 MIT dissertation; what changed was running it across a cluster.
Order transactions by time, and pick how to know the time
A decentralized commit needs a global order without asking one coordinator about everything. Spanner pays for GPS and atomic clocks and waits out a few milliseconds of uncertainty; CockroachDB accepts NTP’s wider error and restarts reads instead; TiDB keeps one timestamp service, simpler but a single point every transaction talks to.
Keep the MySQL or PostgreSQL wire protocol
New databases lose existing drivers, admin and reporting tools, and trained people. Speaking an existing protocol lets applications and ORMs connect unchanged, even when the engine underneath supports only part of the original’s features.

Timeline

  1. 2011

    Matthew Aslett of The 451 Group coins “NewSQL”

  2. 2012

    Google’s Spanner paper describes externally consistent distributed transactions

  3. 2016

    Pavlo and Aslett ask “What’s Really New with NewSQL?”

  4. 2017

    Cloud Spanner opens to the public; CockroachDB 1.0 ships

  5. 2025

    Amazon Aurora DSQL becomes generally available

Sources23

Improve this page on GitHub