Monitors and dashboards as code - Strategy

Use this page for a monitor audit, a new service's monitors, a dashboard, or cleaning up downtimes. The principles behind it are in alert and monitor hygiene. The API names below are illustrative. Swap the URL, headers and field names for your monitoring tool.

Credentials

Use a read-only API credential for audits. Keep a separate, reviewed credential for applies in CI.
Keys live in a secret manager or environment variables loaded from one. Never in scripts, tickets or prompts.
Use the endpoint for your site or region.

Export (read-only)

Export monitors and downtimes to JSON with your tool's CLI or API, then keep the files in the audit working directory.

: "${MONITORING_API_KEY:?load from secret manager}"
API=https://api.example-monitoring.com

curl -sS -H "Authorization: Bearer $MONITORING_API_KEY" \
  "$API/monitors" > monitors.json

curl -sS -H "Authorization: Bearer $MONITORING_API_KEY" \
  "$API/downtimes" > downtimes.json

Audit with jq

Field names vary. Map these checks onto whatever your export uses for id, name, state, tags, message and schedule.

# Monitors in No Data (or the equivalent stale state)
jq -r '.[] | select(.state=="No Data") | [.id, .name] | @tsv' monitors.json

# Monitors with no team tag
jq -r '.[] | select((.tags | map(startswith("team:")) | any) | not) | [.id, .name] | @tsv' monitors.json

# Monitors that notify nobody
jq -r '.[] | select(.message | test("@[A-Za-z0-9_-]") | not) | [.id, .name] | @tsv' monitors.json

# Active downtimes with no end, any scope
jq -r '.[] | select(.status=="active" and .schedule.end == null) | [.id, .scope] | @tsv' downtimes.json

Sort the report in the order from the hygiene page: endless wildcard downtimes, No Data on critical paths, broken handles, unowned, noisy.

Manage as code

Define each monitor in IaC with your tool's provider, next to the service it watches.

resource "monitoring_monitor" "api_errors" {
  name    = "[<service>] 5xx rate high in <env>"
  type    = "query alert"
  query   = "sum:errors{service:<service>,env:<env>} / sum:hits{service:<service>,env:<env>} > 0.05"
  message = <<-EOT
    5xx rate above 5% for <service> in <env>.
    Runbook: https://wiki.example.internal/runbooks/<service>
    @<team-oncall-handle>
  EOT
  critical = 0.05
  warning  = 0.02
  notify_no_data    = true
  no_data_timeframe = 20
  tags = ["team:<team>", "service:<service>", "env:<env>"]
}

Every monitor and dashboard for a service lives with that service's IaC or in a shared monitoring repo.
Import existing UI-made monitors before changing them.
Downtimes for planned work are created with an end time, by the pipeline or by the person doing the work.

Dashboards that help on-call

Top row: the SLO and its burn rate, request rate, error rate, latency (p50, p95, p99).
Second row: saturation (CPU, memory, connections, queue depth).
Third row: dependencies (database, cache, downstream API errors).
Deploy markers overlaid, so changes line up with graphs.
Template variables for env and service, not one dashboard per environment.

Pitfalls

API credentials tied to a person. When they leave, automation breaks. Use a service account.
Muting from the UI during an incident and never unmuting.
High-cardinality tags (user IDs, request IDs) in custom metrics. Expensive and slow.