Background jobs with a queue
The API drops slow work onto SQS and returns at once; a pool of Fargate workers drains the queue at a steady pace.
Click any box to open its page. Open it in the playground to rewire it, change the traffic or take parts down.
Some work is too slow to do while a user waits: resizing a video, generating a PDF, calling a flaky partner API. The API puts a message describing the job on a queue and responds right away. Workers pull jobs when they have capacity, so a sudden burst piles up in the queue instead of overwhelming anything, and a crashed worker’s job simply becomes visible again for another worker.
How it flows
- 1
The API enqueues and returns
For each upload, the API sends a small message such as
{"job": "thumbnail", "key": "uploads/123.jpg"}to SQS and replies202 Accepted. The user isn’t kept waiting for the processing. - 2
Workers pull when they’re free
Each worker long-polls the queue, takes a job, and the message becomes invisible to others for the visibility timeout. During a burst the queue absorbs the excess instead of the workers.
- 3
Success deletes the message
When the thumbnail is written to S3, the worker deletes the message. If the worker crashes first, the message reappears after the timeout and another worker retries it.
- 4
Scale on queue depth
An autoscaling rule watches how many messages are waiting and adds workers when the backlog grows, then removes them when it’s empty.
Why this shape works
- Users get fast responses regardless of how slow the job is
- Bursts are smoothed out: workers run at a steady rate instead of falling over
- Retries come for free, and a dead-letter queue catches jobs that keep failing
- API and workers scale and deploy independently
Where it strains
- Standard SQS queues can deliver a message more than once; make jobs idempotent
- Set the visibility timeout longer than your slowest job, or jobs run twice
- Users need a way to learn the job finished: polling, a webhook or a notification
- A growing backlog is a silent failure; alert on the age of the oldest message
Common swaps
Short jobs (under 15 minutes)? Let Lambda consume the queue directly and skip managing workers.
Multi-step jobs with retries per step and long waits fit a workflow engine better than a bare queue.
Built from
AWS LambdaAWSServerless functions (FaaS)AWS’s functions service: upload a handler, connect a trigger, and pay per request and per millisecond.
Amazon SQSAWSMessage queuesAWS’s fully managed message queue: no brokers to run, billed per request.
AWS FargateAWSContainersRuns ECS tasks and EKS pods without servers: you size each task, AWS finds the machine.
Amazon S3AWSObject storageAWS’s object storage, and the de facto API most object stores copy.