---
title: Monitors and dashboards as code
slug: monitors-and-dashboards
summary: How to export, audit and manage monitors, downtimes and dashboards as code, with read-only recipes an agent can run.
order: 20
updated: 2026-11-11
status: ready
---

Use this page for a monitor audit, a new service's monitors, a dashboard, or cleaning up downtimes. The principles behind it are in [alert and monitor hygiene](./alert-and-monitor-hygiene.md). The API names below are illustrative. Swap the URL, headers and field names for your monitoring tool.

## Credentials

- [ ] Use a read-only API credential for audits. Keep a separate, reviewed credential for applies in CI.
- [ ] Keys live in a secret manager or environment variables loaded from one. Never in scripts, tickets or prompts.
- [ ] Use the endpoint for your site or region.

## Export (read-only)

Export monitors and downtimes to JSON with your tool's CLI or API, then keep the files in the audit working directory.

```bash
: "${MONITORING_API_KEY:?load from secret manager}"
API=https://api.example-monitoring.com

curl -sS -H "Authorization: Bearer $MONITORING_API_KEY" \
  "$API/monitors" > monitors.json

curl -sS -H "Authorization: Bearer $MONITORING_API_KEY" \
  "$API/downtimes" > downtimes.json
```

## Audit with jq

Field names vary. Map these checks onto whatever your export uses for id, name, state, tags, message and schedule.

```bash
# Monitors in No Data (or the equivalent stale state)
jq -r '.[] | select(.state=="No Data") | [.id, .name] | @tsv' monitors.json

# Monitors with no team tag
jq -r '.[] | select((.tags | map(startswith("team:")) | any) | not) | [.id, .name] | @tsv' monitors.json

# Monitors that notify nobody
jq -r '.[] | select(.message | test("@[A-Za-z0-9_-]") | not) | [.id, .name] | @tsv' monitors.json

# Active downtimes with no end, any scope
jq -r '.[] | select(.status=="active" and .schedule.end == null) | [.id, .scope] | @tsv' downtimes.json
```

Sort the report in the order from the hygiene page: endless wildcard downtimes, No Data on critical paths, broken handles, unowned, noisy.

## Manage as code

Define each monitor in IaC with your tool's provider, next to the service it watches.

```hcl
resource "monitoring_monitor" "api_errors" {
  name    = "[<service>] 5xx rate high in <env>"
  type    = "query alert"
  query   = "sum:errors{service:<service>,env:<env>} / sum:hits{service:<service>,env:<env>} > 0.05"
  message = <<-EOT
    5xx rate above 5% for <service> in <env>.
    Runbook: https://wiki.example.internal/runbooks/<service>
    @<team-oncall-handle>
  EOT
  critical = 0.05
  warning  = 0.02
  notify_no_data    = true
  no_data_timeframe = 20
  tags = ["team:<team>", "service:<service>", "env:<env>"]
}
```

- [ ] Every monitor and dashboard for a service lives with that service's IaC or in a shared monitoring repo.
- [ ] Import existing UI-made monitors before changing them.
- [ ] Downtimes for planned work are created with an end time, by the pipeline or by the person doing the work.

## Dashboards that help on-call

- [ ] Top row: the SLO and its burn rate, request rate, error rate, latency (p50, p95, p99).
- [ ] Second row: saturation (CPU, memory, connections, queue depth).
- [ ] Third row: dependencies (database, cache, downstream API errors).
- [ ] Deploy markers overlaid, so changes line up with graphs.
- [ ] Template variables for `env` and `service`, not one dashboard per environment.

## Pitfalls

- API credentials tied to a person. When they leave, automation breaks. Use a service account.
- Muting from the UI during an incident and never unmuting.
- High-cardinality tags (user IDs, request IDs) in custom metrics. Expensive and slow.