---
title: Incident triage
slug: incident-triage
summary: How an agent takes the first fifteen minutes of an incident, from alert to a clear status, a likely cause and a human-approved mitigation.
order: 2
updated: 2026-11-11
status: ready
---

Use this page when an alert fires, a user reports something broken, or a dashboard looks wrong. The goal of triage is not the root cause. The goal is to answer three questions fast: how bad is it, what changed, and what is the safest way to stop the bleeding. Root cause comes later, in the [postmortem](./postmortems.md).

Follow the [agent operating rules](./agent-operating-rules.md) throughout. During an incident they matter more, not less.

## Declare first, investigate second

If customers are affected, or might be, an incident exists. Say so early and in one place.

- [ ] Open or find the incident channel or ticket. One place, linked from everywhere else.
- [ ] Name a human incident commander. An agent can scribe and investigate; it does not command.
- [ ] Post a first status within 5 minutes, even if it says "investigating, impact unknown".

```text
[INCIDENT] <service> elevated errors since ~<HH:MM UTC>
Impact: <who/what is affected, or "unknown, checking">
Status: investigating
Next update: <HH:MM UTC>
IC: <name>   Scribe: <agent or name>
```

## Size the impact with numbers

Pick a severity from facts, not feelings.

| Severity | Typical signal | Response |
| --- | --- | --- |
| SEV1 | Core flow down for most users, data loss, security breach | Page everyone needed, updates every 15 min |
| SEV2 | Core flow degraded, or one region or customer segment down | Page owning team, updates every 30 min |
| SEV3 | Non-core feature broken, workaround exists | Ticket, fix in business hours |
| SEV4 | Cosmetic or internal-only | Ticket |

Impact is severity times users affected, added up over time.

$$I \approx \int_{t_0}^{t_1} \text{severity}(t) \cdot \text{users}(t)\,dt \qquad I = \text{duration} \times \text{severity} \times \text{users}$$

For example, 45 minutes at 40% severity across 12,000 users is \(45 \times 0.4 \times 12{,}000 = 216{,}000\) user-minutes. Use the right-hand form for a quick estimate.

- [ ] Error rate and latency for the affected endpoint, compared with the same hour yesterday.
- [ ] Which environments, regions and tenants are affected. Is it everyone or a slice?
- [ ] Is the error budget for the related [SLO](./slos-and-error-budgets.md) burning fast?

## Find what changed

Most incidents follow a change. Check the last few hours before the first bad data point.

- [ ] Deploys: pipeline history for the service and its dependencies.
- [ ] Config and feature flags: recent flag flips, parameter changes.
- [ ] Infrastructure: Terraform applies, scaling events, AWS Health events.
- [ ] Certificates and secrets: anything that expired or rotated (see [certificate renewal](./certificate-renewal-and-expiry.md)).
- [ ] Upstream vendors: status pages for DNS, CDN, identity providers, and other dependencies.

```bash
# Who changed what in the last 3 hours (read-only)
aws --profile <readonly-profile> --region <region> cloudtrail lookup-events \
  --start-time "$(date -u -d '-3 hours' +%FT%TZ)" \
  --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false \
  --query 'Events[].{t:EventTime,who:Username,what:EventName,src:EventSource}' \
  --output table

# Recent ECS deployments for a service
aws --profile <readonly-profile> --region <region> ecs describe-services \
  --cluster <cluster> --services <service> \
  --query 'services[0].deployments[].{status:status,td:taskDefinition,created:createdAt,rollout:rolloutState}'
```

More read-only recipes are in [AWS read-only investigation](./aws-read-only-investigation.md).

## Narrow it down with a decision tree

```mermaid
flowchart TD
  A[Alert or report] --> B{Customers affected?}
  B -- no --> C[SEV3/4 ticket, keep watching]
  B -- yes --> D{Recent change in the window?}
  D -- yes --> E[Propose rollback of that change]
  D -- no --> F{Dependency unhealthy?}
  F -- yes --> G[Check vendor or upstream status, fail over if designed to]
  F -- no --> H{Capacity or saturation?}
  H -- yes --> I[Propose scale-out]
  H -- no --> J[Gather logs and traces, escalate to service owner]
```

## Mitigate with the smallest reversible step

Mitigation beats diagnosis. Prefer, in order:

1. Roll back the last change (see [blue-green deploys with rollback](./blue-green-deploys-with-rollback.md)).
2. Turn off the feature flag.
3. Scale out or shift traffic away from the bad instance, zone or region.
4. Fail over to the standby.
5. Only then, a forward fix.

- [ ] Write the proposed action, its blast radius and how to undo it.
- [ ] Get the incident commander's explicit yes for this specific action.
- [ ] Do it, or hand the exact command to the human who will.
- [ ] Watch the key metric for 10 minutes before declaring it mitigated.

## Verify and hand off

- [ ] Error rate and latency back to the baseline you recorded earlier.
- [ ] Synthetic checks or smoke tests pass.
- [ ] Status update: "mitigated, monitoring", with the time.
- [ ] Timeline captured: first signal, detection, declare, mitigate, resolve (UTC).
- [ ] Postmortem ticket opened and linked, owner named.

## Pitfalls

- Retrying a failed mitigation blindly. Stop, read the output, report.
- Fixing forward in production while a one-click rollback exists.
- Debugging in a private DM. Everything goes in the incident channel.
- Pasting log lines that contain tokens, card numbers or customer data into the channel or a model. Redact first.
- Treating "no alerts" as "healthy". Check that the monitor itself is not in No Data or muted (see [alert and monitor hygiene](./alert-and-monitor-hygiene.md)).