Cloud Parts

Workflow orchestration

A durable state machine that drives multi-step processes to completion.

6 technologies on this page ↓
Live model
What you’re watchingEach order moves through the steps. About one card charge in four is declined, and those orders take the compensation branch instead of shipping.

How it works

You define the steps of a process, their order, and what happens on failure: retries with backoff, timeouts, branches, parallel steps and compensating actions. The engine records every transition, so a process can wait days for a person or an outside system and resume exactly where it left off, even after a crash.

Think of it as

A project manager with a checklist who never forgets where each project stands, chases late steps, and follows the backup plan when something fails.

Reach for it when

  • Processes that span services: order fulfilment, onboarding, data pipelines
  • Long waits on people, approvals or callbacks
  • Replacing tangled chains of queues and cron jobs that nobody can debug

Think twice when

  • One quick step; a function or queue is enough
  • Very high-rate, low-latency event processing

A real example

An order workflow validates the cart, tries to charge the card up to 3 times, then ships. If the charge is declined, it runs a compensation step that releases the reserved stock and tells the customer.

Technologies

Specific products, and what each does differently.

AWS Step FunctionsAWSAWS’s serverless workflow engine: state machines in JSON that call AWS services directly.Cloudflare WorkflowsCommercialDurable execution on Cloudflare Workers: a workflow is ordinary TypeScript or Python code split into steps, and each step’s result is saved so a failed run retries that step and carries on instead of starting over.PrefectOpen sourceA Python workflow orchestrator where a pipeline is an ordinary function decorated with `@flow`, with retries, caching, schedules and a UI added around it.TemporalOpen sourceAn open-source durable execution engine: you write workflows as ordinary code and they survive crashes.Amazon EventBridge SchedulerAWSEventBridge’s serverless scheduler: millions of one-time or recurring schedules, each of which calls an AWS API at a set time in any time zone.Mainly Event busesDagsterOpen sourceA Python data orchestrator built around assets: you declare the tables, files and models that should exist, and Dagster works out what to run.Mainly ETL & ELT pipelines

What each provider calls it

AWS
Step Functions; EventBridge Scheduler for timed starts
Google Cloud
Workflows
Azure
Durable Functions, Logic Apps
Open source & others
Temporal, Apache Airflow, Argo Workflows; for data pipelines, Dagster (now part of Prefect)

In code · python

# Temporal workflow: each step is retried and its result recorded
@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order: Order) -> str:
        await workflow.execute_activity(validate, order, start_to_close_timeout=T(seconds=30))
        try:
            await workflow.execute_activity(charge_card, order,
                retry_policy=RetryPolicy(maximum_attempts=3),
                start_to_close_timeout=T(minutes=2))
        except ActivityError:
            await workflow.execute_activity(release_stock, order, start_to_close_timeout=T(seconds=30))
            return "cancelled"
        await workflow.execute_activity(ship, order, start_to_close_timeout=T(days=3))
        return "shipped"

Worth exploring

Article· June 2026Durable Queue Workers With Just PostgresFatih Yildiz (mfyz)Starts with the FOR UPDATE SKIP LOCKED queue, then shows what to add when a job has several steps: Absurd stores each step’s result in Postgres, so a crashed worker resumes from the last checkpoint. The author calls it a fit for small and medium projects, not high volume.
Article· June 2026Making Postgres Queues ScaleQian Li and Peter Kraft, DBOSHow DBOS got its Postgres-backed queues to 30,000 workflows a second: SKIP LOCKED, the weaker READ COMMITTED isolation unless a queue needs global limits, partial indexes on waiting rows only, and an index that matches the dequeue order. DBOS wrote it about its own product, but the techniques apply to any Postgres queue.
All resources →

Often confused with

Container orchestrationMessage queuesEvent busesActor modelETL & ELT pipelinesContainer or workflow orchestration?
Where it came from1987–2026

Business processes have always spanned many steps and long stretches of time: an order is checked, paid for, packed and shipped; an insurance claim moves from desk to desk. In 1987 Hector Garcia-Molina and Kenneth Salem of Princeton described “sagas”: a long-lived transaction written as a sequence of smaller ones, with compensating transactions to undo a partial run. Their examples were monthly bank statements and insurance claims.

The 1990s brought workflow management systems, and the Workflow Management Coalition, founded in 1993, worked to standardize them. IBM and Microsoft merged their web-service orchestration languages, WSFL and XLANG, into BPEL4WS, an XML language that they, BEA, SAP and Siebel submitted to the OASIS standards body in 2003. Cloud providers and web companies then brought the idea to developers: Amazon’s Simple Workflow Service in 2012, Airbnb’s Apache Airflow for data pipelines in 2015, AWS Step Functions in 2016, and Uber’s Cadence, which became Temporal, where workflows are ordinary code.

Today the idea often goes by “durable execution”: the engine records every step, so a process can wait days for a person or survive a crash and resume where it stopped. AI agents, which run long chains of model and tool calls, have renewed interest; Temporal raised $550 million in 2026 as demand grew for reliable AI infrastructure.

Why it’s built this way

One engine owns the whole process
A central orchestrator knows every step and where each instance stands, so failures, timeouts and stuck steps show up in one place instead of being scattered across queues and cron jobs.
Compensation instead of long locks
Holding database locks for a process that runs hours or days blocks everything else, so sagas commit each step and undo completed ones with compensating actions if a later step fails.
Record every state transition
Writing each step down durably lets the engine resume after a crash, and lets a process sleep for days without holding a server.
Definitions as data, or workflows as code
BPEL and Step Functions describe the process as XML or JSON that an engine can validate and draw. Cadence and Temporal use a normal programming language instead, which requires deterministic code so the engine can replay it.

Timeline

  1. 1987

    The sagas paper introduces compensating transactions

  2. 2003

    BPEL4WS, from IBM and Microsoft, is submitted to OASIS

  3. 2012

    Amazon Simple Workflow Service launches

  4. 2015

    Airbnb releases Airflow as open source

  5. 2016

    AWS Step Functions launches

  6. 2020

    Temporal 1.0, a fork of Uber’s Cadence, ships

  7. 2026

    Temporal raises $550M as demand for reliable AI infrastructure grows

Sources13

Improve this page on GitHub