Incident management

When something big breaks: declare it, give one person command, keep everyone informed, stop the damage, then learn from it without blame.

1 technology on this page ↓

How it works

An incident is an unplanned problem big enough to need coordinated work, such as an outage, data loss or a security breach. Teams set the bar in advance, often with severity levels: SEV1 when a customer-facing service is down for everyone, down to minor bugs that can wait for working hours. Anyone can declare one, and declaring early is cheaper than scrambling hours later. Declaring opens a channel and a shared document and assigns roles. The incident commander doesn’t debug; they keep the overall picture, hand out work and make decisions. An operations lead and responders change the system, and only they do. A communications lead posts regular updates to stakeholders and to a public status page. The first goal is mitigation, not understanding: roll back the deploy, drain the bad region, turn off the feature, and find the root cause once users are no longer hurting. When service is restored the incident is resolved, and serious ones get a written postmortem: a timeline, the impact, what made the failure possible and what made it hard to fix, and follow-up actions with owners. A blameless postmortem asks how the system let a reasonable person make that mistake, not who to punish, because people who fear blame hide the details that prevent the next one.

Think of it as

A fire crew at a burning building. One chief directs, firefighters fight the fire, a spokesperson talks to the press and the neighbours, and nobody investigates the cause until the fire is out; afterwards the investigators look for what failed, not for someone to fire.

Reach for it when

Customer-visible outages or degradations
Problems that need a second team, or are still unsolved after an hour
Data loss, security events, and anything leadership or customers will ask about

Think twice when

Heavyweight process for a small, single-person fix; a ticket is enough
Hunting for the root cause while users are still affected
Postmortems that name a culprit; people stop reporting what really happened

A real example

At 10:12 an alert shows 40% of logins failing. The on-call engineer declares a SEV1, opens #inc-login-errors and takes incident command; a second engineer becomes ops lead and a support manager handles communications. The status page says “Investigating” at 10:15. The ops lead sees the failures began with a 10:05 config change and rolls it back at 10:24; errors drop to normal and the status page says “Monitoring”. The incident is resolved at 10:40. Three days later a blameless postmortem finds the config change skipped the canary stage, and adds a check that makes that impossible.

Used for incident management

Incident management is something products do, not a kind of product. These are often used for it, grouped by what they are.

Observability →

Better StackCommercialUptime checks, on-call paging and status pages, plus logs, traces, metrics and errors in a ClickHouse-based store, sold as one SaaS billed per responder and per GB.


Better Stack	Observability	Commercial product	Better Stack, Inc.	Proprietary (the Better Stack collector is Apache 2.0)	2026-10-05

Roles in an incident#

Google’s SRE book borrows these roles from the Incident Command System. In a small incident one person may hold several; the commander holds every role not yet handed out.

Incident commander

Holds the high-level state, decides priorities and assigns work. Removes roadblocks; doesn’t debug.

Operations lead

Ops

Works with responders to apply fixes. The operations team should be the only group changing the system during the incident.

Communications lead

The public face of the response: periodic updates to stakeholders and the status page, and keeps the incident document current.

Planning lead

longer term

Files bugs, arranges handoffs, orders food, and tracks how the system has diverged from normal so it can be put back.

Severity levels, an example#

Atlassian publishes the scale it uses. The exact cut-offs matter less than agreeing on them before an incident.

Level	Meaning	Example
SEV 1	Critical incident with very high impact	A customer-facing service is down for all customers; customer data loss
SEV 2	Major incident with significant impact	A customer-facing service is down for a subset of customers
SEV 3	Minor incident with low impact	A bug is a minor inconvenience to customers

At Atlassian, SEV 1 and SEV 2 page on-call engineers at any hour; lower severities can wait for working hours.

Often confused with

On-call Alerting Error tracking

Where it came from1970–2021

The model comes from firefighting. Over 13 days from 22 September to 4 October 1970, 17 major wildfires in southern California burned half a million acres, destroyed nearly 700 structures and killed 16 people. A 1980 Forest Service report lists what went wrong between the agencies: no central source of up-to-date facts, no central planning, radio channels jammed with traffic, and different terms, organisation structures and procedures from one agency to the next. Federal, state and local agencies built FIRESCOPE in response, and with it the Incident Command System: one commander, standard roles and a common vocabulary. In 1982 it was broadened to all kinds of incidents, and after a 2003 presidential directive it became part of the US National Incident Management System.

Software operations borrowed it. Google’s SRE book bases the company’s incident process on ICS, with an incident commander, an operations lead, a communications lead and a planning role, a live incident document and clear rules for when to declare. Its list of best practices starts with priorities: “Stop the bleeding, restore service, and preserve the evidence for root-causing.” The incident chapter of the 2018 SRE Workbook, written with engineers from PagerDuty, adds that first responders “must prioritize mitigation above all else”.

Learning afterwards changed too. In 2012 John Allspaw described Etsy’s blameless postmortems and its aim for a “Just Culture”: engineers give a detailed account of what they did and why, “without fear of punishment or retribution”, because blame hides exactly the details needed to prevent a repeat. Google’s book notes that blameless culture came from healthcare and avionics. Measurement got the same scrutiny: Google’s Štěpán Davidovič showed with simulations that mean time to recovery (MTTR) is poorly suited to tracking improvement, the 2021 VOID report found incident durations too skewed for averages to mean much, and DORA has since replaced MTTR with failed deployment recovery time.

Why it’s built this way

One person in command who isn’t fixing: In the 1970 fires, and in many outages since, everyone worked hard and nobody held the whole picture. A commander who doesn’t debug can coordinate, communicate and notice when the plan isn’t working.
Mitigate before you understand: Users don’t care why it broke, only that it works again. Generic mitigations such as rollback, draining traffic or disabling a feature restore service while the cause is still unknown.
Declare early and often: Spinning up the process for something that turns out small costs little; starting it hours into a growing outage costs a lot. Clear triggers, such as a second team being needed or customer-visible impact, remove the hesitation.
Blameless postmortems: People who expect punishment give a sanitised story. Asking how the system made the mistake easy produces the details that lead to real fixes.

Timeline

1970
Southern California fires expose failures of multi-agency coordination
1980
Forest Service report describes FIRESCOPE and the Incident Command System
1982
ICS broadened to all hazards in the National Interagency Incident Management System
2003
US presidential directive leads to the National Incident Management System
2012
John Allspaw on blameless postmortems and a Just Culture at Etsy
2016
Google’s SRE book describes incident command for software
2021
The VOID report questions MTTR as a metric

Sources6

Improve this page on GitHub