cloudparts

Chaos engineering

Break parts of your system on purpose, in a controlled experiment, to learn whether it really survives the failures it was built to survive before a real outage tests it for you.

6 technologies on this page ↓

How it works

A chaos experiment is run like a science experiment. First you define the steady state: a measurable output that shows the system is healthy, such as orders per minute, error rate or latency, rather than anything about its internals. Then you state a hypothesis: this number will stay where it is while something breaks. Then you inject a fault that mimics a real event (a crashed server, a severed network connection, a slow dependency, a full disk, throttled cloud API calls, a lost Availability Zone) and look for a difference between the part of the system you broke and a control group you didn’t. If the steady state holds, you have more confidence; if it doesn’t, you found a weakness before your users did. Tools inject faults at different layers. Some call the cloud provider’s API to stop instances or force a database failover. Some run an agent on each host or Kubernetes node that stresses CPU and memory, kills processes, shifts the clock or adds latency and packet loss to a container’s network. Some sit in the network path as a proxy, and some live inside the application as an SDK or extension that delays or fails individual requests. Because the faults are real, safety comes first: start in staging, then in production on one instance or a few percent of traffic (a small blast radius), and wire in a stop condition, usually an alarm on the steady-state metric, that halts the experiment and reverts the fault. Teams often start with game days, planned sessions where people watch an experiment together, and then automate experiments to run on a schedule or in the delivery pipeline, so a regression in resilience shows up the week it ships.

Think of it as

A fire drill. You set off the alarm on a quiet Tuesday morning with everyone at their desks, one floor at a time, so you find the blocked exit and the warden who doesn’t know the route then, not at night during a real fire.

Reach for it when

  • Systems built to tolerate failure, with replicas, retries, timeouts, circuit breakers and failover, where you need evidence that those mechanisms work today and not just on the design diagram
  • Before a known peak such as a sale or a launch, to check autoscaling, failover and fallbacks under the failures you expect
  • After an incident, to reproduce the failure on purpose and confirm the fix holds
  • Checking the people side: an experiment also tests whether alerts fire, dashboards show the problem and the runbook is right

Think twice when

  • You can’t see the steady state yet: without metrics and alerts you can’t tell whether an experiment hurt users, or stop it in time
  • Weaknesses you already know about: if the database has no replica, an experiment only confirms the outage; fix it first
  • Faults that can’t be undone in production, such as shutting down hosts or killing processes that don’t restart by themselves, until recovery is automatic
  • Surprising the owners of the system: an experiment nobody agreed to is an incident

A real example

A shop’s checkout page calls a recommendations service and is meant to show bestsellers if that call takes longer than 300 ms. The team’s hypothesis: if recommendations get 2 seconds slower, checkout completions per minute stay within 2% of normal. In staging they add 2 s of network latency to one recommendations pod with Chaos Mesh, then to all of them. Checkout slows to over 2 seconds: the 300 ms timeout lives in a config key that the new HTTP client no longer reads. They fix it, rerun the experiment, and the steady state holds. A month later the same experiment runs in production against 5% of pods, with an alarm on checkout errors as the stop condition, and then weekly on a schedule.

Technologies

Specific products, and what each does differently.

What each provider calls it

AWS
AWS Fault Injection Service (FIS): experiment templates with actions on EC2, ECS, EKS, RDS, Lambda, networking and more, CloudWatch alarms as stop conditions, and prebuilt scenarios such as an Availability Zone power interruption
Google Cloud
No managed fault injection service; Google Cloud publishes chaos engineering guidance and recipes built on the open-source Chaos Toolkit
Azure
Azure Chaos Studio: service-direct faults through Azure APIs and agent-based faults inside VMs, with Chaos Mesh for AKS; its newer Workspaces and Scenarios model is in public preview
Open source & others
Chaos Mesh and LitmusChaos (CNCF incubating, Apache 2.0) for Kubernetes; Chaos Toolkit; Toxiproxy (MIT) for network faults in tests; Netflix’s Chaos Monkey (Apache 2.0). Commercial: Gremlin, Harness Chaos Engineering

Where the fault is injected#

Each layer can break different things, and each needs different access to your systems.

LayerWhat it can breakExamples
Cloud APIStop or terminate instances, fail over a database, pause replication, throttle API calls, cut a zone’s networkAWS FIS, Azure Chaos Studio service-direct faults, Chaos Monkey
Host or node agentCPU, memory and disk pressure, killed processes, clock skew, latency and packet loss on a container’s networkChaos Mesh’s Chaos Daemon, LitmusChaos, the Gremlin agent, FIS through Systems Manager
Proxy in the network pathLatency, timeouts, dropped or sliced connections between an app and one dependencyToxiproxy, Gremlin Failure Flags sidecar
Inside the applicationErrors or delays on chosen requests, users or code pathsGremlin Failure Flags SDKs, the FIS extension for Lambda

In code · yaml

# Chaos Mesh: one random pod of web-show gets 10 ms of extra latency
# for 12 seconds, at minute 5 of every hour
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: schedule-delay-example
spec:
  schedule: '5 * * * *'
  historyLimit: 2
  concurrencyPolicy: 'Forbid'
  type: 'NetworkChaos'
  networkChaos:
    action: delay
    mode: one
    selector:
      namespaces: [default]
      labelSelectors:
        app: web-show
    delay:
      latency: '10ms'
    duration: '12s'

Worth exploring

Article· February 2024Vault chaos engineeringKalam Abdul, HashiCorpFour chaos experiments on a three-node Vault cluster with integrated storage, run with Gremlin under load from the Vault benchmark tool: losing the leader (a standby takes over with little effect), losing quorum (reads and writes stop until you follow the recovery runbook), added latency and memory pressure. A concrete worked example of hypothesis, steady-state metrics and abort conditions; written by HashiCorp with help from Gremlin, so the fixes it suggests lean on Vault Enterprise features such as performance standbys and DR clusters.
Video· April 2021Automating for Failure: Disaster Recovery Testing with Terraform & GremlinSamson Olufuwa and Olubusayo Amowe (Fuerza), published by HashiCorpA 27-minute introductory talk on disaster recovery: the common strategies (backup and restore, pilot light, warm standby, multi-region), RTO and RPO, and how keeping infrastructure in Terraform modules and AMIs lets you rebuild it quickly, with a demo on AWS and regular drills to test the plan. The slides cover Terraform far more than Gremlin, and say plainly that Terraform does not back up your data; published on HashiCorp’s site, so it favours Terraform.
Article· April 2018Automating a Chaos Engineering Environment on AWS with Terraform and GremlinHo Ming Li, GremlinA step-by-step tutorial: Terraform builds a throwaway VPC and one EC2 instance whose user data installs the Gremlin agent, you run a latency attack from the Gremlin web app and watch ping times rise by about 100 ms, then terraform destroy removes it all. Useful as a pattern for disposable chaos test environments; it is Gremlin’s own tutorial, last updated in 2018 and flagged by Gremlin as possibly out of date, and it hardcodes agent credentials in user data, which it says not to do long term.
All resources →

Often confused with

Automated testingSynthetic monitoringIncident managementCanary releases
Where it came from2010–2022

Breaking things on purpose started inside large web companies. Jeff Barr traces it to work by “Master of Disaster” Jesse Robbins in the early days of Amazon.com, and Google ran its own disaster recovery testing program, DiRT. The name and the habit spread from Netflix. In December 2010, describing its move to AWS, Netflix wrote that “the best way to avoid failure is to fail constantly” and that one of the first systems its engineers built there was the Chaos Monkey, whose job was “to randomly kill instances and services within our architecture”.

In July 2011 Netflix described a Simian Army of tools built on the same idea, including Latency Monkey, which added delays between services, and Chaos Gorilla, which simulated the loss of an entire Availability Zone. It open-sourced Chaos Monkey in July 2012, by which point the tool had terminated over 65,000 instances in a year. Netflix engineers then turned the practice into a discipline: the Principles of Chaos Engineering defined it as “the discipline of experimenting on a system in order to build confidence” in its ability to withstand turbulent conditions in production, and a 2016 paper in IEEE Software by Ali Basiri, Casey Rosenthal and colleagues laid out the method.

Products followed. Gremlin, founded in 2016 by Kolton Andrus, who had built fault injection at Amazon and worked on reliability at Netflix, came out of stealth in December 2017. Kubernetes brought open-source platforms that describe faults as custom resources: LitmusChaos, started in 2017, and PingCAP’s Chaos Mesh, open-sourced at the end of 2019; both are now CNCF incubating projects. AWS made its Fault Injection Simulator generally available in March 2021, and Microsoft followed with Azure Chaos Studio.

Why it’s built this way

Experiments, not tests
A test checks a behaviour you already specified. The Principles frame chaos as an experiment that tries to disprove a hypothesis about the steady state, so it can find failure modes nobody thought to write a test for.
Run in production, during business hours
Netflix ran Chaos Monkey on weekdays between 9 a.m. and 3 p.m. so engineers would be at work to respond, and the Principles argue that only real traffic exercises the real request paths, which no staging environment reproduces.
Measure outputs, not internals
Throughput, error rate and latency show whether users are affected. Internal signals such as CPU can move without harm, so the hypothesis is stated in terms of what customers see.
Keep the blast radius small and automate
Experiments cause real harm when the hypothesis is wrong, so they start with one instance or a slice of traffic and stop on an alarm. Automation then keeps checking as the system changes, since a fix made last month can be undone by this week’s deploy.

Timeline

  1. 2010

    Netflix describes Chaos Monkey as one of the first systems it built on AWS

  2. 2011

    Netflix announces the Simian Army, including Chaos Gorilla for losing a whole Availability Zone

  3. 2012

    Chaos Monkey is open-sourced in July

  4. 2016

    “Chaos Engineering” paper in IEEE Software; Gremlin is founded

  5. 2017

    LitmusChaos starts; Gremlin launches its product in December

  6. 2019

    PingCAP open-sources Chaos Mesh on December 31

  7. 2021

    AWS Fault Injection Simulator becomes generally available in March

  8. 2022

    LitmusChaos and Chaos Mesh become CNCF incubating projects

Sources14

Improve this page on GitHub