Sub-millisecond lookups

Below a millisecond, the data has to live in memory and the request has to be simple: get or set one key, bump a counter, push onto a list. In-memory stores such as Redis and Memcached do the work itself in microseconds or less, so most of the time is spent on the network and the operating system. Redis’s own docs put a typical 1 Gbit/s network round trip at about 200 µs, and a busy virtual machine can add milliseconds of its own. Pipelining, batching several keys per request and keeping connections open matter more than the server’s raw speed.

What breaks low latency is anything that blocks the server: a command that walks a large collection, a fork for a snapshot on a big instance, swapping, or an fsync on every write. A cache in front of a database, such as DynamoDB Accelerator, can bring reads from single-digit milliseconds down to microseconds, but its reads are only eventually consistent and it helps only when most requests hit the cache.

When you compare systems, look at p99 and p99.9 latency measured from the client in the same zone, under the target load, with persistence and replication switched on as they would be in production. Check how it behaves while a replica is resyncing or a snapshot is being written, and whether the client library supports pipelining.