cloudparts

NoSQL databases

The umbrella name for databases that drop the relational model and SQL, usually to spread data over many machines: key-value, document, wide-column and graph stores, and a few others.

3 technologies on this page ↓

How it works

NoSQL isn’t one design but a family of them, grouped by data model. Key-value stores keep an opaque value under each key. Document databases store JSON-like records they can index and query field by field. Wide-column stores keep huge tables split into partitions, with rows sorted inside each. Graph databases store nodes and the edges between them. Most share a few traits from the systems that started the wave in the late 2000s, Google’s Bigtable and Amazon’s Dynamo. Data is partitioned by a key, by hashing or by key ranges, so a cluster grows by adding machines. Each partition is copied to several nodes, and many of these systems let a write succeed on some replicas and reconcile the rest later, which keeps them available during failures but means a read can briefly return old data. There is no fixed schema to migrate, and in place of joins you usually store data the way it will be read, often copying it into several shapes. Many later added what they first left out: secondary indexes, SQL-like languages such as Cassandra’s CQL, multi-document transactions in MongoDB, strong consistency modes. The term covers anything that isn’t relational, so it says little on its own; the useful question is which data model and which consistency a system offers.

Think of it as

“Not a restaurant” describes a food truck, a bakery and a vending machine equally well. Each feeds people faster in its own niche, but you still have to ask what it sells.

Reach for it when

  • Data or traffic that will outgrow one server, where you can design around a partition key
  • Access patterns known in advance: lookups by ID, a user’s recent items, a document read whole
  • Data that fits a non-tabular shape better: nested documents, graphs of connections, sparse wide rows
  • Writes that must keep succeeding during network or datacenter failures

Think twice when

  • Ad-hoc questions, reports and joins across many entities; a relational database or a warehouse answers those
  • Invariants across many records, such as balances and stock, that need transactions everywhere; consider a relational or distributed SQL database
  • Data that fits on one machine: PostgreSQL with jsonb covers many document use cases with less to learn

A real example

A messaging app keeps sessions in a key-value store, user profiles in a document database, and each conversation’s messages in a wide-column table partitioned by conversation and sorted by time. Each store answers its one kind of question in milliseconds at any size. A question such as “how many messages did users in Norway send last month?” fits none of them, so the app also copies events to a data warehouse.

Technologies

Specific products, and what each does differently.

What each provider calls it

AWS
DynamoDB (key-value and document); DocumentDB (MongoDB-compatible); Amazon Keyspaces (Cassandra-compatible wide-column); Neptune (graph); ElastiCache and MemoryDB (in-memory key-value)
Google Cloud
Firestore (documents); Bigtable (wide-column); Memorystore (in-memory key-value)
Azure
Azure Cosmos DB, one managed service for document, key-value, graph, table and vector data, with its own NoSQL API and compatible APIs for other databases
Open source & others
Key-value: Valkey, Redis, etcd, FoundationDB; document: MongoDB (SSPL), CouchDB; wide-column: Cassandra, ScyllaDB (source-available), HBase; graph: Neo4j, JanusGraph; multi-model: ArangoDB, SurrealDB (both BSL)

The four main families#

Each family has its own page. They differ in what the database understands about your data, and so in what it can answer without reading everything.

What is storedFast to answerExamples
Key-valueAn opaque value under each keyGet or put by keyValkey, Redis, DynamoDB, Aerospike
DocumentJSON-like documents in collectionsQueries and indexes on any field, whole-object readsMongoDB, CouchDB, Firestore
Wide-columnRows in partitions, sorted by a clustering keyWrites, and range reads within one partitionCassandra, ScyllaDB, Bigtable, HBase
GraphNodes and typed edges, both with propertiesWalking relationships several hops deepNeo4j, Neptune, JanusGraph
  • Several products span families: DynamoDB is key-value and document, Cosmos DB serves document, key-value, graph and table data from one service, and ArangoDB and SurrealDB mix documents and graphs.

Often confused with

Key-value storesDocument databasesWide-column storesGraph databasesRelational databasesNewSQL and distributed SQL
Where it came from1998–2012

The word first named something else: in 1998 Carlo Strozzi called his small relational database NoSQL because it was used from the Unix shell instead of through SQL. The modern sense comes from two papers. Google’s Bigtable (2006) described a sorted, distributed map of rows and columns spread over thousands of machines, and Amazon’s Dynamo (2007) a key-value store that stayed writable through failures by giving up immediate consistency. Open-source projects copied both: HBase and Hypertable followed Bigtable, Voldemort and Dynomite followed Dynamo, and Cassandra, from Facebook, combined them.

On 11 June 2009 Johan Oskarsson of Last.fm organised a meetup in San Francisco on open-source, distributed, non-relational databases, with talks on Voldemort, Cassandra, Dynomite, HBase, Hypertable, CouchDB, MongoDB and others. Asked on IRC for a short name, Eric Evans suggested “NoSQL”, and it stuck. Evans later regretted that the vague name grouped projects that solved very different problems, and Neo4j’s Emil Eifrem pushed the reading “Not only SQL”.

The label spread fast through the early 2010s, then blurred as the systems converged with relational ones. NoSQL databases added query languages, secondary indexes and transactions, relational databases added JSON columns, and distributed SQL databases kept SQL while scaling out like their NoSQL neighbours. Clouds now sell these engines as “purpose-built” databases, one per data model.

Why it’s built this way

Partition by key and scale out
Web companies outgrew what one database server could hold or serve. Splitting data by a key across commodity machines let capacity grow by adding nodes instead of buying bigger ones.
Relax consistency for availability
Dynamo’s shopping cart had to accept writes even when disks failed or the network split. Letting replicas diverge and reconcile later kept writes flowing, at the cost of reads that can be briefly stale.
No fixed schema, no joins
Joins across machines are expensive. Storing data in the shape it is read, a whole document or a pre-sorted partition, keeps every query on one node and lets fields change without migrations.

Timeline

  1. 1998

    Carlo Strozzi names his shell-based relational database NoSQL

  2. 2006

    Google publishes the Bigtable paper

  3. 2007

    Amazon publishes the Dynamo paper

  4. 2009

    The NoSQL meetup in San Francisco on 11 June gives the movement its name

  5. 2012

    AWS launches DynamoDB

Sources9

Improve this page on GitHub