Cloud Parts

Prefect

A Python workflow orchestrator where a pipeline is an ordinary function decorated with @flow, with retries, caching, schedules and a UI added around it.

What it is
Workflow orchestration,ETL & ELT pipelines
Type
Open source
Made by
Prefect Technologies
License
Apache 2.0; Prefect Cloud is proprietary
Facts checked
9 October 2026
Website Source code

What it is

In Prefect you decorate Python functions with @flow and @task. A flow is the workflow; tasks are the steps inside it, each with its own retries, timeouts and caching. There is no separate DAG file: loops, if statements and calls happen at run time as in any Python program, and Prefect records each flow run and task run as it moves through states such as Scheduled, Running, Completed, Failed or Crashed. .submit() and .map() run tasks concurrently, and a cache key built from a task’s inputs, its code and the flow run lets a retried flow skip steps that already succeeded. Prefect 3 made events and automations open source, so a flow can start in response to an event, and assets track what flows produce.

To run on a schedule or remotely, a flow becomes a deployment. The simplest way is flow.serve(), a long-running Python process that runs the flow on a cron or interval. For more control, deployments go to work pools: a worker, a lightweight process you run, polls the pool and starts each flow run as a subprocess, a Docker container, a Kubernetes Job, an ECS task or a Cloud Run container. Push pools hand runs straight to ECS, Cloud Run, Azure Container Instances or Modal with no worker, and Prefect Managed pools run them on Prefect’s machines. The control plane is either a self-hosted Prefect server (API, UI and a database, SQLite by default or PostgreSQL 14.9+ in production) or Prefect Cloud, which adds SSO, role-based access and audit logs. Cloud has a free Hobby tier (2 users, 5 deployments), Starter at $100 a month and Team at $100 per user a month (checked October 9, 2026).

Why teams pick it

  • Pipelines are plain Python: existing functions gain retries, logging and state tracking with two decorators, and dynamic loops or branches need no DAG rewrite
  • Flexible execution: run in a local process with serve, on your own Kubernetes, Docker or ECS through workers, or on serverless push and managed pools without a worker
  • Events and automations are open source since 3.0, so event-driven runs and alerts don’t require Prefect Cloud
  • Prefect Cloud prices by users and deployments rather than per run, with your code and data staying in your own infrastructure on hybrid work pools

Watch out for

  • Major versions have broken user code: Prefect 1.0 flows needed changes to run on 2.0 (2022), and 3.0 (2024) moved to Pydantic 2 and removed async tasks in sync flows
  • Caching across runs needs result persistence, which is off by default, so results must be stored somewhere such as S3 or GCS
  • Self-hosting means running the server, its database and workers yourself, and SSO, RBAC and audit logs exist only in Prefect Cloud
  • Prefect announced in July 2026 that it is acquiring Dagster Labs, so one company now steers two rival orchestrators; it positions Prefect for execution and Dagster for data assets

Alternatives

What all workflow orchestration have in common →
AWS Step FunctionsAWSAWS’s serverless workflow engine: state machines in JSON that call AWS services directly.Cloudflare WorkflowsCommercialDurable execution on Cloudflare Workers: a workflow is ordinary TypeScript or Python code split into steps, and each step’s result is saved so a failed run retries that step and carries on instead of starting over.TemporalOpen sourceAn open-source durable execution engine: you write workflows as ordinary code and they survive crashes.

How it fits with other technologies

A flow with a retrying task and plain Python control flow · python

import random
from prefect import flow, task

@task(retries=3, retry_delay_seconds=1)
def fetch_orders(day: str) -> list[int]:
    if random.random() < 0.5:                 # a flaky API
        raise ConnectionError("API timed out")
    return [120, 80, 50]

@task
def total(amounts: list[int]) -> int:
    return sum(amounts)

@flow(log_prints=True)
def daily_revenue(days: list[str]):
    for day in days:                          # plain Python loops and ifs, no DAG file
        amounts = fetch_orders(day)
        if amounts:
            print(f"{day}: {total(amounts)}")

if __name__ == "__main__":
    daily_revenue(["2026-10-07", "2026-10-08"])
    # daily_revenue.serve(name="nightly", cron="0 2 * * *")  # schedule it instead

# One run’s log (abridged):
# Flow run 'auburn-albatross' - 2026-10-07: 250
# Task run 'fetch_orders-660' - Task run failed with exception: ConnectionError('API timed out')
#   - Retry 1/3 will start 1 second(s) from now
# Task run 'fetch_orders-660' - Finished in state Completed()
# Flow run 'auburn-albatross' - 2026-10-08: 250
# Flow run 'auburn-albatross' - Finished in state Completed()

Prefect, Dagster and Temporal#

All three run steps and retry failures, but they start from different questions. Prefect’s own founder describes the split: Dagster’s answer was assets, Prefect’s was execution. Temporal records every step so code can resume exactly where it stopped.

PrefectDagsterTemporal
What you writeFlows and tasks: decorated Python functions with normal control flowAssets: the tables, files and models that should existWorkflows as code, with side effects in activities
What it tracksFlow and task runs, their states, events and assetsAssets, their dependencies and each materializationThe event history of each workflow execution
After a failureTask retries; cached results let a rerun skip finished stepsRun retries re-execute from the failed step by defaultReplays the history on another worker and continues
Runs that wait a long timeAutomations start new runs on eventsSensors react to events and start new runsOne execution can last seconds or years
LicenceApache 2.0; Prefect CloudApache 2.0; Dagster+ (owned by Prefect since 2026)MIT; Temporal Cloud
  • Prefect’s comparison pages, including Prefect vs Airflow, are the vendor’s own view.

See it in a real stack

No architecture uses Prefect yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2018–2026

Prefect was founded in 2018 by Jeremiah Lowin, when, as Lowin later wrote, “most of the world’s data work still ran on cron jobs and hand-tended DAGs”. Its open-source Python package reached PyPI in March 2019. The pitch, still on Prefect’s comparison pages, is that a pipeline should be your Python code as-is, not code restructured into DAG objects, operators and XCom messages the way Apache Airflow requires.

Prefect 2.0 left beta on July 27, 2022 after nearly a year of building in public, and it was a rewrite: Prefect 1.0 flows needed changes to run on it. Prefect 3.0 followed in September 2024 with an open-source events and automations backend, faster runtime, and caching built on transactional semantics. On July 13, 2026, Lowin announced that Prefect was acquiring Dagster Labs, its rival since both were founded in 2018. Prefect says Dagster and Dagster+ continue, and describes the split as Dagster defining outcomes, Prefect executing the work and FastMCP, its third product family, governing access for AI agents.

Why it’s built this way

Workflows as ordinary Python, not a declared DAG
Tasks are created as the code runs, so a flow can loop over data or branch on a result without describing the graph up front, which Prefect presents as its main difference from Airflow’s static DAGs.
Orchestration separate from execution
Prefect Cloud or a Prefect server tracks state and schedules, while workers and push pools run the code in the user’s own infrastructure, so data doesn’t have to pass through Prefect.
Workers polling typed work pools
Workers replaced the older agents during Prefect 2 and are standard in Prefect 3. A pool’s type (Kubernetes, Docker, ECS and so on) decides how each run’s infrastructure is created.
Transactional caching in 3.0
Cache keys hash a task’s inputs, code and run, so retrying a pipeline reuses results that already succeeded instead of redoing their side effects.

Timeline

  1. 2018

    Jeremiah Lowin founds Prefect

  2. 2019

    First release of the open-source prefect package on PyPI, in March

  3. 2022

    Prefect 1.0 in February; Prefect 2.0 leaves beta in July

  4. 2024

    Prefect 3.0 in September with open-source events and automations

  5. 2026

    Prefect announces it is acquiring Dagster Labs in July

Sources20 · facts checked 9 October 2026

Improve this page on GitHub