Cloud Parts

Key-value stores

Put a value under a key and get it back by that key, fast, at almost any scale.

10 technologies on this page ↓
Live model
What you’re watchingEach request’s key is hashed to one partition, so the same key (same color) always lands on the same partition, and load spreads across all three.

How it works

A key-value store keeps a map from keys to values and offers little more than three operations: put a value under a key, get it back, and delete it. The database never looks inside the value, which can be a string, a serialized object or an image. That keeps every operation a lookup: microseconds for in-memory stores such as Valkey, Redis and MemoryDB, single-digit milliseconds for DynamoDB on disk. To scale out, the key is hashed to pick a partition, so data and traffic spread across many servers and the cluster grows by adding more. The price is that the key is the only way in. You can’t search on what’s inside a value, and there are no joins, so you choose keys around the questions you’ll ask. Many stores add a little on top: Redis and Valkey offer lists, sets, sorted sets and atomic counters as values, DynamoDB sorts items within a partition and can index other attributes, and etcd adds transactions and watches on keys.

Think of it as

A wall of numbered lockers. Give the number and you have the bag in seconds; ask for every bag with a red scarf inside and someone has to open every locker.

Reach for it when

  • Sessions, shopping carts, feature flags and rate-limit counters
  • Lookups by ID at huge scale, with steady low latency
  • Configuration and coordination data a cluster must agree on, like Kubernetes state in etcd

Think twice when

  • Finding records by what’s inside them; use a document database or add a search index
  • Reporting, ad-hoc questions and joins

A real example

A web shop keeps each shopper’s session and cart in Valkey under session:<id>, set to expire after 30 minutes. Every page view is one GET, so any of the shop’s servers can handle the next click. Finding every cart that holds a given product, though, would mean reading every key.

Technologies

Specific products, and what each does differently.

Amazon DynamoDBAWSAWS’s serverless key-value and document database, with single-digit-millisecond reads at any size.Amazon ElastiCacheAWSAWS’s managed Valkey, Redis OSS and Memcached, as node-based clusters or a serverless cache.Amazon MemoryDBAWSA Valkey- and Redis-compatible database that keeps everything in memory but loses no acknowledged write.FoundationDBOpen sourceAn ordered, distributed key-value store with strictly serializable ACID transactions across the whole cluster, built as a small core that other databases are layered on.MemcachedOpen sourceA simple, multithreaded in-memory key-value cache; clients hash keys across independent servers.RedisOpen sourceAn in-memory data-structure server: the default cache, and also queues, counters and leaderboards.RocksDBOpen sourceThe embedded key-value engine Facebook forked from LevelDB: an LSM tree in a C++ library that many databases and stream processors use as their storage layer.ValkeyOpen sourceThe Linux Foundation’s BSD-licensed fork of Redis 7.2.4: an in-memory key-value server that Redis clients can talk to unchanged.Workers KVCommercialCloudflare’s global key-value store that keeps data in a few central stores and caches it in the data centers where it is read, so reads are fast and writes take up to a minute to show up everywhere.etcdOpen sourceThe Raft-replicated key-value store that holds every Kubernetes cluster’s state, built for small, critical data that many machines must agree on.Mainly Coordination services

What each provider calls it

AWS
DynamoDB; ElastiCache and MemoryDB for Valkey or Redis OSS
Google Cloud
Memorystore for Valkey or Redis; Bigtable for durable lookups by row key
Azure
Azure Managed Redis; Azure Cosmos DB for Table
Open source & others
Valkey (BSD), Redis (AGPLv3 option since 8.0), etcd, FoundationDB, TiKV; embedded in your program: RocksDB

See it in a real stack

Architectures that use key-value stores

Serverless web app on AWSCloudFront serves the site from S3 and sends API calls through API Gateway to Lambda functions backed by DynamoDB.Amazon CloudFrontAmazon S3Amazon API GatewayAWS LambdaAmazon DynamoDBWeb app on KubernetesA load balancer spreads requests over app pods on Kubernetes, which check Redis before going to PostgreSQL.AWS Application Load BalancerKubernetesRedisPostgreSQL

In code · bash

# valkey-cli (redis-cli works the same)
SET session:9f3a '{"user":42,"cart":["sku-17"]}' EX 1800   # expires in 30 minutes
GET session:9f3a

# an atomic counter per user per minute, for rate limiting
INCR rate:42:0930
EXPIRE rate:42:0930 60

Worth exploring

WebsiteDB-EnginesDB-Engines, provided by Redgate SoftwareA directory of hundreds of database systems with their data models, licences and features, which you can compare side by side. Its monthly ranking measures popularity (search results, Google Trends, Stack Overflow questions, job ads, LinkedIn profiles, posts on X), not quality or speed.
All resources →

Often confused with

Document databasesWide-column storesCachingRelational databasesCoordination servicesLSM treesKey-value, document or wide-column?
Where it came from1979–2024

Storing values under keys is one of the oldest ideas in data storage. Ken Thompson’s dbm library, released with Seventh Edition Unix in 1979, kept key-value pairs in files, and Berkeley DB, first released in 1991, offered the same model as a library that programs could embed. But data shared by many users mostly lived in relational databases.

Web-scale services broke that habit. In 2007 Amazon described Dynamo, a key-value store for services like the shopping cart that only looked data up by primary key and had to keep accepting writes even when disks failed or networks broke. In 2009 Salvatore Sanfilippo released Redis, a server that kept everything in memory and let values be lists, sets and counters as well as strings. In 2012 AWS turned the Dynamo lessons into DynamoDB, a managed service.

Key-value stores also became the foundation under other systems. Kubernetes keeps all of its cluster state in etcd, and FoundationDB, which Apple open-sourced in 2018, adds ACID transactions across keys. In 2024, after Redis moved to source-available licences, the Linux Foundation forked it as Valkey, which AWS and Google Cloud now run as managed services.

Why it’s built this way

The value is opaque
If the database never parses values, every operation is a lookup by key. That keeps latency low and predictable, and a value can be anything from a counter to an image.
Hash keys to spread data across machines
Dynamo used consistent hashing so a cluster could grow one node at a time, moving only a small share of the data. Hash partitioning is still how these stores scale.
Availability before immediate consistency
A customer should always be able to add to a cart, even during failures. Dynamo never rejected a write because of a failure and reconciled diverging copies later, the trade-off now called eventual consistency.
Keep everything in memory
Redis held the whole dataset in RAM so every operation was fast, and wrote it to disk only for durability. Memory limits the size, but it made data structures like sorted sets cheap to offer.

Timeline

  1. 1979

    Ken Thompson’s dbm key-value library ships with AT&T Unix

  2. 1991

    First general release of Berkeley DB

  3. 2007

    Amazon publishes the Dynamo paper

  4. 2009

    Redis is released as open source

  5. 2012

    AWS launches DynamoDB as a managed service

  6. 2018

    Apple open-sources FoundationDB

  7. 2024

    Redis changes licence; the Linux Foundation forks Valkey

Sources20

Improve this page on GitHub