Canary releases

Send a small share of real traffic to the new version, compare it with the old one, and widen the share only while it looks healthy.

Live modeldata or requestresponsebusy / idle instance

What you’re watchingThe router sends one request in ten to the canary (orange) and the rest to the stable version. Real canaries often start smaller, at 1% or 5%, and the share grows only while the canary’s metrics match the stable version’s.

How it works

The new version (the canary) runs next to the current one (the control, or stable version). A router splits traffic by weight: perhaps 1% or 5% to the canary at first, the rest to stable. Then you wait and compare the two: error rate, latency, CPU, and business numbers such as checkouts per minute. If the canary is no worse, the weight goes up in steps, say 5%, 25%, 50%, 100%; if it is worse, the weight goes back to 0% and only that small slice of users ever saw the problem. The split can happen at a load balancer or API gateway, in a service mesh, in a function alias, or simply by running 1 new instance out of 10. Comparing by eye is slow and unreliable, so larger teams automate it: tools such as Kayenta, Argo Rollouts and Flagger query the metrics, score the canary against the control, and promote or roll back on their own. Two versions serve at once for the whole rollout, so the same backward-compatibility rules as a rolling deployment apply. Canaries are not A/B tests: a canary looks for regressions and should finish in minutes or hours, while an A/B test compares variants to test a hypothesis and can take days to reach statistical significance. Feature flags can run a canary too, by turning code on for a 1% cohort of users instead of routing requests to different servers.

Think of it as

Miners carried a canary in a cage down the pit. If toxic gas leaked in, the bird fell ill long before the miners did, and everyone got out. The small slice of traffic is the bird.

Reach for it when

High-traffic services where even a few minutes of a bad release would hurt many users
Changes whose real behaviour only shows under production load and real data
You have metrics good enough to tell a healthy version from a sick one within minutes

Think twice when

Low traffic: 1% of a few requests a minute gives no signal; use blue-green with good tests instead
No way to split traffic or compare versions; without measurement a canary is just a slow rollout
Software installed on users’ devices, where you don’t control when each copy upgrades

A real example

A payments API on Lambda publishes version 58 and points its live alias 95% at version 57 and 5% at 58. After 10 minutes the dashboards show the canary’s p99 latency at 310 ms against 120 ms for stable, so the deploy job sets the canary weight back to 0. About 5% of requests for 10 minutes saw slow responses, instead of everyone. The next attempt, with the fix, goes 5%, 25% and then 100% over half an hour.

What each provider calls it

AWS: Lambda weighted aliases; CodeDeploy canary configurations (e.g. `LambdaCanary10Percent5Minutes`, `ECSCanary10Percent5Minutes`); API Gateway canary release on a stage; Amazon ECS canary and linear deployments
Google Cloud: Cloud Run traffic splitting between revisions (`gcloud run services update-traffic --to-revisions LATEST=5`)
Azure: App Service slot traffic routing (Traffic %); Container Apps traffic splitting between revisions
Open source & others: Argo Rollouts (`setWeight` steps, analysis), Flagger, Spinnaker with Kayenta automated canary analysis

Canary, feature flag, A/B test or blue-green?#

They use similar plumbing, so they get mixed up. The question each answers is different.

Canary release

is the new version safe?

Route a small, growing share of requests to a new deployment and compare its health with the old one. Done in minutes or hours, then everyone is on the new version.

Feature flag rollout

is this feature safe?

Ship the code to everyone, switched off, and turn it on for a cohort such as 1% of users. Works per feature rather than per deployment, and needs no traffic routing, at the cost of flags to clean up later.

A/B test

which variant is better?

Show two variants to comparable groups and measure behaviour, such as sign-ups. An experiment that can run for days until the result is statistically significant.

Blue-green deployment

can we switch back instantly?

Move all traffic from one full environment to another at once. No gradual exposure, but rollback is a single switch.

In code · yaml

# Argo Rollouts: the canary gets more traffic step by step
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments
spec:
  replicas: 20
  selector:
    matchLabels: { app: payments }
  template:
    metadata:
      labels: { app: payments }
    spec:
      containers:
        - name: payments
          image: registry.example.com/payments:58
  strategy:
    canary:
      steps:
        - setWeight: 5          # 1 of 20 pods without a traffic router
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: {}             # wait for a human to promote
# kubectl argo rollouts abort payments    # send everything back to stable

Often confused with

Blue-green deployments Rolling deployments Load balancers Observability API gateways Rolling, blue-green or canary?

Where it came from2009–2018

The name comes from coal mining, where miners carried a caged canary underground as an early warning: toxic gas would kill the bird before it killed the miners. In software it means exposing a new version to a small part of production first, so problems show up before they reach everyone.

IMVU, a company practising continuous deployment, automated this early. In 2009 Timothy Fitz described its push script: the new code went live on a small subset of machines, and a minute later the script compared load, errors and other metrics across the cluster, rolling back automatically on a statistically significant regression before pushing to 100%. Jez Humble and David Farley covered canary releasing in Continuous Delivery (2010), and in 2014 Danilo Sato’s entry on Martin Fowler’s site gave the general recipe and warned against mixing canaries with A/B tests.

The comparison step was the hard part to do by hand. In April 2018 Netflix and Google open-sourced Kayenta, which pulls metrics for the canary and the control, runs statistical tests and returns a score, as part of Spinnaker. That August, RedMonk’s James Governor proposed the name progressive delivery for canarying, feature flags and A/B testing at scale; Kubernetes tools such as Argo Rollouts and Flagger now automate weighted traffic shifting and metric analysis under that banner.

Why it’s built this way

Expose a small, random share first: With a small random sample, most users never see a bad version, and no single user is affected all the time. It also tests the new version under real production load instead of in a separate capacity-testing environment.
Compare against a control running at the same time: Comparing with last week’s numbers mixes in time of day and traffic changes. Running stable and canary side by side, and comparing the two, isolates the effect of the release.
Automate the judgement: Teams like Google and Netflix compare deployments many times a day, which manual analysis can’t keep up with. Statistical tests on chosen metrics give a consistent pass or fail that can promote or roll back without a human.

Timeline

2009
IMVU describes automatic canary pushes with statistical rollback
2010
Humble and Farley describe canary releasing in Continuous Delivery
2014
Danilo Sato’s CanaryRelease entry on martinfowler.com
2018
Netflix and Google open-source Kayenta in April; “progressive delivery” is coined in August

Sources14

Improve this page on GitHub