Monitors as code with Terraform: resource, trio auth, explicit no-data

# Monitors as code with Terraform

Click-ops monitors rot: nobody knows which are current, thresholds drift, and the incident review finds the monitor that should have fired was edited by someone who left. Terraform fixes the audit trail, not just the creation.

## Setup

Provider needs the trio: `api_key`, `app_key`, `api_url` (the api_url must match your site). In CI, feed them from env vars backed by the secret manager, never from committed files.

```hcl
resource "datadog_monitor" "high_error_rate" {
  name    = "[team] checkout-api high error rate"
  type    = "metric alert"
  query   = "avg(last_5m):avg:trace.express.request.errors{service:checkout-api,env:prod} by {service} > 0.05"
  message = "Error rate breached. Runbook: [link]. @slack-alerts"
  monitor_thresholds {
    critical          = 0.05
    critical_recovery = 0.01
  }
  notify_no_data    = true
  renotify_interval = 60
  tags = ["team:checkout", "service:checkout-api"]
}
```

## Discipline

1. **notify_no_data is a decision, not a default.** Set it explicitly on every monitor. For critical monitors you usually want true (silent data loss is an incident).
2. **Name with the team first.** `[team] service: what` makes the alert list scannable at 3am.
3. **Message carries the runbook.** The alert message should link the runbook and tag the responder channel. A monitor without a next step is a notification, not an alert.
4. **Import existing monitors** before writing new ones (the docs cover importing resources into Terraform) so apply does not duplicate or fight click-ops monitors.
5. **Ship apply events** with dogwrap (`pip install datadog`, then wrap the apply) so monitor changes show up in the event stream next to deploys.

## Verify

`terraform plan` on a no-op change shows no diff (drift detection is the point). Trigger a test alert in staging and confirm the message, tags, and routing land where you expect.