Cloud Parts

Dagster

A Python data orchestrator built around assets: you declare the tables, files and models that should exist, and Dagster works out what to run.

What it is
ETL & ELT pipelines,Workflow orchestration
Type
Open source
Made by
Prefect Technologies (acquired Dagster Labs, formerly Elementl, in 2026)
License
Apache 2.0; Dagster+ (the hosted service) is proprietary
Facts checked
6 October 2026
Website Source code

What it is

In Dagster you write Python functions decorated with @asset. Each one produces a table, file or model, and its parameters name the assets it reads, so Dagster builds an asset graph of what depends on what. Running a function and storing its result is a materialization. Assets can be split into partitions, such as one per day or per region, so a backfill only reruns the slices you pick. Schedules, sensors (functions that check for new files or events and request runs) and declarative automation conditions such as “update whenever an upstream asset changes” decide when things run. Integrations load whole tools into the graph: each dbt model, Fivetran or Airbyte table and Sling replication becomes an asset with its own history.

A self-hosted deployment has three long-running services. The webserver serves the UI and a GraphQL API, the daemon evaluates schedules and sensors and takes runs off the queue, and code location servers load your definitions in their own processes. The run launcher starts each run as a subprocess, a Docker container, a Kubernetes Job or an ECS task, and runs, event logs and schedule state go to SQLite by default or PostgreSQL or MySQL in production. Dagster+ hosts the control plane for you: with Hybrid your code runs on an agent in your own cloud with no compute charge; with Serverless it runs on Dagster’s machines at $0.01 a minute. Plans start at $10 a month (Solo, one user) or $100 a month (Starter, up to three users), plus $0.040 or $0.035 per credit, where every asset materialization and every op run is one credit (prices checked October 6, 2026).

Why teams pick it

  • The asset graph shows lineage, freshness and the last run of every table, which a list of task runs doesn’t
  • Partitions and backfills are built in: rerun one day, one region or a whole range, as one run per partition or one run for the range
  • Model-level dbt integration: each dbt model is its own asset and dbt tests become asset checks
  • Assets are plain Python functions, so you can run and unit-test them locally before deploying
  • The open-source core is Apache 2.0, and run launchers and storage are swappable, from a laptop with SQLite to Kubernetes with PostgreSQL

Watch out for

  • Dagster+ bills per materialization: 100 hourly-partitioned assets make 72,000 credits a month, about $2,520 on Starter before any compute
  • Prefect announced it was buying Dagster Labs in July 2026 and the founder left; Prefect says Dagster keeps its name and licence, but the long-term roadmap is now Prefect’s
  • Self-hosting means running the webserver, a single daemon (it can’t be replicated) and a PostgreSQL or MySQL database
  • Built for batch data pipelines in Python, not for one run that waits days on a person; that needs durable timers and signals like Temporal’s
  • Large partition sets slow the UI; Dagster recommends at most 100,000 partitions per asset

Alternatives

What all etl & elt pipelines have in common →
AirbyteCommercialAn ELT platform you can host yourself: hundreds of connectors, each a Docker image speaking one JSON protocol, scheduled by services on Kubernetes.Apache NiFiOpen sourceA Java dataflow server you program by drawing a graph of processors in a web UI, with buffered queues, back pressure and a record of where every piece of data went.ArtieCommercialA managed change data capture service that copies every insert, update and delete from production databases into a warehouse, aiming for under a minute of delay.BruinOpen sourceOne Go command-line tool that ingests data, runs SQL and Python transformations and checks the results, with each step a file in your repo.dbtOpen sourceSQL `select` statements in git, built into warehouse tables in dependency order, with tests and docs alongside.EstuaryCommercialA managed data-movement platform that captures changes from databases and SaaS sources into collections kept as JSON files in your own bucket, then keeps warehouses and other systems in sync, in real time or on a schedule.FivetranCommercialManaged connectors that copy data from SaaS apps, databases and files into your warehouse or lake, billed by the rows that change each month.MeltanoOpen sourceA command-line ELT tool that runs Singer taps and targets from a `meltano.yml` project kept in git.OLakeOpen sourceAn open-source tool in Go that copies databases, Kafka topics and S3 files into Apache Iceberg tables (or plain Parquet), with full loads, incremental syncs and CDC, and no Kafka, Debezium, Spark or Flink in between.

How it fits with other technologies

Two daily-partitioned assets: one on a schedule, one that follows it · python

import dagster as dg
import pandas as pd

daily = dg.DailyPartitionsDefinition(start_date="2026-01-01")

@dg.asset(partitions_def=daily)
def raw_orders(context: dg.AssetExecutionContext) -> pd.DataFrame:
    day = context.partition_key  # e.g. "2026-10-05"
    return pd.read_parquet(f"s3://acme-raw/orders/{day}.parquet")

# the parameter name makes raw_orders an upstream dependency
@dg.asset(partitions_def=daily, automation_condition=dg.AutomationCondition.eager())
def daily_revenue(raw_orders: pd.DataFrame) -> pd.DataFrame:
    return raw_orders.groupby("country", as_index=False)["amount"].sum()

orders_job = dg.define_asset_job("orders_job", selection=[raw_orders], partitions_def=daily)

defs = dg.Definitions(
    assets=[raw_orders, daily_revenue],
    schedules=[dg.build_schedule_from_partitioned_job(orders_job, hour_of_day=2)],
)

# dagster dev -f pipeline.py  (UI and daemon on http://localhost:3000)

Dagster, Apache Airflow and Temporal#

All three run steps in order and retry failures, but they track different things. Dagster tracks data assets, Airflow tracks task runs in DAGs, and Temporal records every step of long-running application code.

DagsterApache AirflowTemporal
Unit you defineAssets: functions that produce a table, file or modelTasks grouped into DAGs; since 3.0, assets as scheduling triggersWorkflows written as code, with side effects in activities
What it knows about dataLineage, partitions and the last materialization of each assetAsset URIs as strings: “makes no assumptions about the content or location of the data”Nothing: it stores the workflow’s event history
Failure handlingOp retries; run retries re-execute from the failed stepTask retries; backfills run inside the scheduler since 3.0Replays the event history on another worker and continues from the same line
Long waits on people or eventsSensors poll and start new runsSensors and event-driven scheduling start new runsDurable timers and signals inside one running workflow
Typical jobBatch and ELT pipelines, dbt projects, ML pipelinesBatch pipelines and scheduled jobsOrder flows, payments, AI agents
Licence and stewardApache 2.0; Prefect (Dagster Labs)Apache 2.0; Apache Software FoundationMIT; Temporal Technologies
Managed optionsDagster+ (Serverless or Hybrid)Amazon MWAA, Google Cloud Composer and othersTemporal Cloud
  • Dagster publishes an Airflow comparison and Airlift, a tool to observe Airflow DAGs from Dagster and move them over gradually; that page is the vendor’s own view.

Dagster+ plans#

List prices checked October 6, 2026. Every asset materialization and every op that runs counts as one credit. Hybrid is on every plan; SSO, SAML and audit logs are Pro only.

SoloStarterPro
Base price$10 a month$100 a monthContact sales
Per credit$0.040$0.035Custom
Serverless compute$0.010 a minute$0.010 a minuteCustom
Users, code locations, deployments1, 1, 1Up to 3, 5, 1Unlimited
Hybrid (your own compute)No compute chargeNo compute chargeCustom
  • Serverless runs get 4 vCPUs, 16 GB of RAM and 128 GB of disk on a single node, with daily caps of 100 GB of bandwidth and 4,500 step-minutes. Heavy or multi-node work belongs on Hybrid.

See it in a real stack

No architecture uses Dagster yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2018–2026

Through the 2010s, many data teams ran their pipelines as what Pete Hunt, a Facebook colleague of Dagster’s founder, called “a rat’s nest of cron jobs and Python scripts”, often put into production with minimal or no tests and monitoring. Nick Schrock, a former Facebook engineer listed among the original contributors to the GraphQL specification, wrote Dagster’s first line of code in 2018 and founded a company, Elementl, to build it. The first release went to PyPI that June under the Apache 2.0 licence.

Dagster Cloud, a hosted control plane, was announced in December 2021. Dagster 1.0 followed on August 5, 2022, with stable APIs and software-defined assets: instead of describing tasks, you describe the tables and files that should exist. Elementl raised a $33 million Series B in May 2023 and renamed itself Dagster Labs that August, because Dagster was its only product.

On July 13, 2026, Schrock announced that Prefect, maker of another Python workflow orchestrator, was acquiring Dagster Labs. About 40 staff joined Prefect and Schrock moved on. Prefect says Dagster keeps its name, its open-source licence and Dagster+.

Why it’s built this way

Assets, not tasks, as the unit of work
Task-based orchestrators show whether pipelines ran, not the state of the tables they produce. Declaring assets in code gives lineage and lets Dagster materialize only what is missing or out of date.
Pipelines as ordinary, testable Python
Data code often went to production untested. Plain functions with typed inputs can be run and unit-tested on a laptop, without a scheduler and workers.
Swappable storage, run launchers and executors
The same definitions run as subprocesses with SQLite on a laptop and as Kubernetes Jobs with PostgreSQL in production; only dagster.yaml changes.
Open core with a hosted control plane
The orchestrator stayed Apache 2.0, and the company sells Dagster Cloud (now Dagster+), whose Hybrid mode keeps customers’ code and data in their own cloud.

Timeline

  1. 2018

    Schrock starts Dagster and Elementl; first PyPI release in June

  2. 2021

    Dagster Cloud announced in December

  3. 2022

    Dagster 1.0 in August, with software-defined assets

  4. 2023

    $33M Series B in May; Elementl becomes Dagster Labs in August

  5. 2026

    Prefect announces it is acquiring Dagster Labs in July; Schrock moves on

Sources32 · facts checked 6 October 2026

Improve this page on GitHub