Cloud Parts

Caching

Keep hot data in memory so you don’t ask the slow system twice.

7 technologies on this page ↓
Live model
What you’re watchingMost requests are answered by the cache (HIT). Misses go through to the slow database, and the answer is stored for next time.

How it works

Caching keeps a copy of hot data in a faster store, usually an in-memory key-value store, in front of something slower, such as a database or an API. On a hit the answer comes back in under a millisecond. On a miss the app fetches from the source and stores the result with a time-to-live. The hard parts are deciding how stale data may get and clearing entries when the source changes.

Think of it as

Keeping the reference book you use every day on your desk instead of walking to the library each time.

Reach for it when

  • Read-heavy data that changes slowly: product pages, config, computed summaries
  • Shielding a database from the same query repeated thousands of times
  • Sessions, rate-limit counters, leaderboards

Think twice when

  • Data that must always be exactly current
  • Being the only copy of anything important; caches get evicted and restarted

A real example

A well-status endpoint takes 180 ms to compute from the database. Caching the result for 60 seconds answers 70% of requests in under a millisecond and cuts database load by the same share.

More on caching

Narrower ideas that build on this one.

Technologies

Specific products, and what each does differently.

Also used for caching

Products of other kinds that are often used this way, grouped by what they are.

Key-value stores →

Amazon ElastiCacheAWSAWS’s managed Valkey, Redis OSS and Memcached, as node-based clusters or a serverless cache.Amazon MemoryDBAWSA Valkey- and Redis-compatible database that keeps everything in memory but loses no acknowledged write.MemcachedOpen sourceA simple, multithreaded in-memory key-value cache; clients hash keys across independent servers.RedisOpen sourceAn in-memory data-structure server: the default cache, and also queues, counters and leaderboards.ValkeyOpen sourceThe Linux Foundation’s BSD-licensed fork of Redis 7.2.4: an in-memory key-value server that Redis clients can talk to unchanged.Workers KVCommercialCloudflare’s global key-value store that keeps data in a few central stores and caches it in the data centers where it is read, so reads are fast and writes take up to a minute to show up everywhere.

What each provider calls it

AWS
ElastiCache, MemoryDB
Google Cloud
Memorystore for Valkey or Redis (Memorystore for Memcached is deprecated; shutdown January 2029)
Azure
Azure Managed Redis (built on Redis Enterprise; no Valkey service)
Open source & others
Redis, Valkey, Memcached

In code · python

import json, redis
r = redis.Redis(host="cache", port=6379)

def well_status(well_id: int) -> dict:
    key = f"well-status:{well_id}"
    if (hit := r.get(key)) is not None:
        return json.loads(hit)                 # fast path
    status = compute_from_database(well_id)    # slow path
    r.set(key, json.dumps(status), ex=60)      # keep for 60 s
    return status

Often confused with

Key-value storesCDNsRelational databasesIn-process cachesCache or database?In-process cache, shared cache or CDN?
Where it came from1968–2024

Caching started in hardware. By the 1960s processors were much faster than main memory, and IBM’s System/360 Model 85, announced in January 1968, became the first commercially available computer with a cache: 16 to 32 KB of “high-speed buffer storage” that cut the effective storage cycle to a third or a quarter of main storage’s. The idea has been reused at every layer since. Keep a small, fast copy of what was used recently, because it will probably be needed again.

The web carried it up to applications. HTTP/1.1, published in January 1997, added the Cache-Control header so browsers and proxies could keep a response for as long as the server allowed. Database-backed sites needed the same trick for their own queries. In May 2003 Brad Fitzpatrick wrote memcached for LiveJournal, which had 2.5 million accounts on about 70 machines. It pooled spare RAM across many servers into one large key-value cache, and LiveJournal reported hit rates around 92%. Facebook and Wikipedia were among the large sites that adopted it.

Redis, from 2009, added data structures and optional persistence. In 2011 AWS launched ElastiCache, a managed service compatible with memcached, so a cache became something you rent rather than run. When Redis changed its licence in 2024, AWS, Google and Oracle backed Valkey, a BSD-licensed fork.

Why it’s built this way

A small, fast store in front of a big, slow one
On the Model 85, a main-storage cycle took about a microsecond, twelve times the processor’s 80-nanosecond cycle, and databases are far slower still. Keeping copies of the most-used data close by avoids most trips to the slow layer.
Hash keys across independent cache servers
Memcached clients treat servers as buckets in a hash table, so each key always goes to the same server. Servers never talk to each other or replicate, which keeps them simple and lets you add capacity by adding machines.
Evict the least recently used items
Memory is limited, so something must go when it fills. Dropping the least recently used entry keeps the data most likely to be asked for again.
Let the source say how long a copy stays fresh
HTTP’s Cache-Control and the time-to-live on a cache key let a copy be served without checking the source each time, trading some staleness for speed.

Timeline

  1. 1968

    IBM announces the System/360 Model 85, the first commercial computer with a cache

  2. 1997

    HTTP/1.1 adds Cache-Control for browsers and proxies

  3. 2003

    Brad Fitzpatrick writes memcached for LiveJournal

  4. 2009

    Redis brings data structures and persistence to in-memory stores

  5. 2011

    AWS launches ElastiCache, a managed memcached-compatible cache

  6. 2024

    Valkey forks Redis under the Linux Foundation after a licence change

Sources11

Improve this page on GitHub