IoT telemetry pipeline
Devices stream readings into Kinesis; Lambda writes them to TimescaleDB for Grafana, and raw data is archived to S3.
Click any box to open its page. Open it in the playground to rewire it, change the traffic or take parts down.
Thousands of sensors each send a reading every few seconds. A stream takes that firehose without blinking and keeps it for a while, so consumers can process it at their own pace. One consumer writes readings into a time-series database for live dashboards; another archives the raw stream to cheap object storage for later analysis.
How it flows
- 1
Devices append to the stream
Each device sends readings with its id as the partition key, so one device’s readings stay in order on one shard. The stream keeps them for at least 24 hours.
DevicesAmazon Kinesis Data Streams - 2
Lambda loads the database
Lambda polls the stream, receives batches of records, and inserts them into a TimescaleDB hypertable that is partitioned by time behind the scenes.
- 3
Dashboards query recent data
Grafana runs time-bucketed SQL such as
time_bucket('1 minute', time)against TimescaleDB every few seconds to draw live charts. - 4
The raw stream is archived
A separate consumer reads the same stream in large batches and writes compressed files to S3, where analysts can query months of history without loading the database.
Why this shape works
- The stream absorbs spikes and lets each consumer run at its own pace
- Adding a consumer (alerting, ML features) doesn’t touch devices or the other consumers
- The time-series database keeps recent data fast to query and old data compressed
- Raw history is kept cheaply and can be replayed if processing logic changes
Where it strains
- Shards cap throughput per partition key range; plan capacity or use on-demand mode
- If the Lambda consumer errors on a bad record, it can block its shard until the record expires
- Retention and downsampling policies decide how much the database grows
- Devices on flaky networks send late or duplicate readings; design inserts to tolerate both
Common swaps
Off AWS or at very high volume, Kafka plays the stream’s role.
For metrics from servers rather than devices, Prometheus scrapes and stores them itself.
For heavy analytical queries over the whole history, load the archive into ClickHouse or an Iceberg table.
Built from
Amazon Kinesis Data StreamsAWSEvent streamsAWS’s serverless event stream, split into shards that each keep records in order.
AWS LambdaAWSServerless functions (FaaS)AWS’s functions service: upload a handler, connect a trigger, and pay per request and per millisecond.
TimescaleDBOpen sourceTime-series databasesA PostgreSQL extension that turns ordinary tables into time-partitioned, compressed time-series tables.
Amazon S3AWSObject storageAWS’s object storage, and the de facto API most object stores copy.
GrafanaOpen sourceObservabilityDashboards and alerts over data that stays where it is: Prometheus, SQL databases, cloud monitoring and more.