Amazon Kinesis Data Streams

AWS’s serverless event stream, split into shards that each keep records in order.

What it is: Event streams
Type: Managed service on AWS
Made by: Amazon Web Services
License: Proprietary
Facts checked: 5 October 2026

What it is

A Kinesis data stream is a set of shards. Each record normally carries a partition key; Kinesis hashes it with MD5 to choose a shard, so records with the same key stay in order. Since September 2026, on-demand streams can also spread records without a key when order doesn’t matter. In provisioned mode each shard accepts up to 1 MB/s or 1,000 records/s of writes and serves 2 MB/s of reads, and you split or merge shards to scale. In on-demand mode Kinesis manages the shards itself: a new stream starts at 4 MB/s of writes and keeps room for twice its peak of the previous 30 days.

Records stay for 24 hours by default, extendable to 365 days, and each can be up to 10 MiB. Ordinary consumers poll with GetRecords and share each shard’s 2 MB/s. Enhanced fan-out consumers get their own 2 MB/s per shard, pushed over HTTP/2 with about 70 ms of delay; a stream allows 20 of them, or 50 with On-demand Advantage in supported Regions. On-demand streams can also deliver records straight to S3, or into Apache Iceberg tables on Amazon S3 Tables, without a consumer application.

Why teams pick it

Serverless: no brokers or disks, and on-demand mode removes shard planning
Consumers can re-read anything inside the retention window, up to 365 days
Plugs into Lambda, Amazon Data Firehose and Managed Service for Apache Flink with no glue code
Built-in delivery to S3 and Iceberg tables replaces a separate loader for the common case

Watch out for

One partition key can’t exceed a single shard’s 1 MB/s or 1,000 records/s, so a hot key throttles
Polling consumers share 2 MB/s and 5 reads per second per shard; several consumers need enhanced fan-out, which costs extra
Its own API, not Kafka’s: Kafka clients and Connect don’t work against it (Amazon MSK is AWS’s Kafka)
Provisioned shards bill by the hour even when idle, and On-demand Standard adds a fixed charge per stream

Alternatives

What all event streams have in common →

Apache FlussOpen sourceStreaming storage built for Flink: event logs and updatable tables in columnar Arrow format, with key lookups, and older data copied into Paimon, Iceberg, Hudi or Lance tables.

Apache KafkaOpen sourceThe open-source distributed log that most event streaming is built on, or built to be compatible with.

Apache PulsarOpen sourceA multi-tenant messaging and streaming platform from Yahoo that keeps brokers stateless and stores messages in Apache BookKeeper, so one topic can serve as a log, a pub-sub feed or a work queue.

Cloudflare K2BetaCommercialCloudflare’s serverless event stream: a durable log stored in R2 that Workers and HTTP clients append to, read through independent subscriptions that lease out batches of records.

RedpandaCommercialA Kafka-compatible streaming broker rewritten in C++ as one binary, with no JVM and no ZooKeeper.

WarpStreamCommercialA Kafka-compatible stream that keeps no data on disks: stateless Agents write straight to your object storage bucket.

How it fits with other technologies

What Amazon Kinesis Data Streams uses

What uses Amazon Kinesis Data Streams

Create an on-demand stream, write a record and read the shard · bash

aws kinesis create-stream --stream-name clicks \
  --stream-mode-details StreamMode=ON_DEMAND

aws kinesis put-record --stream-name clicks --partition-key user-42 \
  --data '{"page": "/pricing"}' --cli-binary-format raw-in-base64-out

ITERATOR=$(aws kinesis get-shard-iterator --stream-name clicks \
  --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON \
  --query ShardIterator --output text)
aws kinesis get-records --shard-iterator "$ITERATOR" --limit 100

# keep a week of history instead of the default 24 hours
aws kinesis increase-stream-retention-period --stream-name clicks --retention-period-hours 168

See it in a real stack

IoT telemetry pipeline →

Live modeldata or requestresponsebusy / idle instance

Open in playground

What you’re watchingDevice readings append to a Kinesis stream. Lambda reads small batches into TimescaleDB, where Grafana’s dashboard queries find them. A second consumer archives the stream to S3 in large, infrequent batches.

Where it came from2013–2025

In 2013 the big-data tools on AWS, such as Hadoop and data warehouses, worked in batches. Companies that needed to act on clickstreams, logs or trades within seconds assembled their own systems from open-source parts, which AWS described as complex, hard to operate and unreliable. AWS’s Terry Hanold put the problem this way: database and MapReduce technologies “are fundamentally batch-based, and struggle with enabling real-time decisions” on a stream of data that never ends.

AWS announced Amazon Kinesis at re:Invent on November 14, 2013 and opened it to all customers that December. Like Apache Kafka, which LinkedIn had open-sourced in 2011, it was a log split into ordered pieces, here called shards, that consumers read at their own pace. Unlike Kafka, it was a managed service with its own API: you chose a number of shards, each taking up to 1 MB/s of writes, and records were kept for 24 hours.

Kinesis grew into a family, starting with Firehose in 2015, and the original service became Kinesis Data Streams. Later releases removed most of its early limits: retention of up to 365 days, an on-demand mode with no shards to plan, and records of up to 10 MiB.

Why it’s built this way

Shards as the unit of capacity and price: Each shard has fixed limits, so throughput, cost and ordering are easy to reason about. The cost was planning: until on-demand mode in 2021, you split and merged shards yourself.
A partition key hashed to pick the shard: Hashing the key with MD5 spreads records across shards while keeping all records for one key, such as one user, in order on one shard.
Reading doesn’t remove records: Records stayed for 24 hours whoever had read them, so several applications could process the same stream and a crashed consumer could pick up where it left off.
Coordination in a client library: The Kinesis Client Library “handles the details of load balancing, coordination, and error handling” on the consumer side, which kept the service itself simple.

Timeline

2013
Announced at re:Invent in November; open to all customers in December
2015
Firehose launches; the original is renamed Kinesis Streams
2018
Enhanced fan-out gives each consumer its own read throughput
2020
Retention can be extended to 365 days
2021
On-demand mode removes shard planning
2025
Records of up to 10 MiB; On-demand Advantage mode

Sources13 · facts checked 5 October 2026

Improve this page on GitHub