Monitors as code with Terraform: resource, trio auth, explicit no-data

TL;DR

For "resource "datadogmonitor" "higherror_rate" {": set it explicitly on every monitor. Terraform fixes the audit trail, not just the creation.

resource "datadog_monitor" "high_error_rate" {
  name    = "[team] checkout-api high error rate"
  type    = "metric alert"
  query   = "avg(last_5m):avg:trace.express.request.errors{service:checkout-api,env:prod} by {service} > 0.05"
  message = "Error rate breached. Runbook: [link]. @slack-alerts"
  monitor_thresholds {
    critical          = 0.05
    critical_recovery = 0.01
  }
  notify_no_data    = true
  renotify_interval = 60
  tags = ["team:checkout", "service:checkout-api"]
}

Steps

1. Click-ops monitors rot: nobody knows which are current, thresholds drift, and the incident review finds the monitor that should have fired was edited by someone who left. Terraform fixes the audit trail, not just the creation.
2. notifynodata is a decision, not a default. Set it explicitly on every monitor. For critical monitors you usually want true (silent data loss is an incident).
3. Name with the team first. [team] service: what makes the alert list scannable at 3am.
4. Message carries the runbook. The alert message should link the runbook and tag the responder channel. A monitor without a next step is a notification, not an alert.
5. Import existing monitors before writing new ones (the docs cover importing resources into Terraform) so apply does not duplicate or fight click-ops monitors.
6. Ship apply events with dogwrap (pip install datadog, then wrap the apply) so monitor changes show up in the event stream next to deploys.
7. terraform plan on a no-op change shows no diff (drift detection is the point). Trigger a test alert in staging and confirm the message, tags, and routing land where you expect.

When to use

You are seeing this: Click-ops monitors rot: nobody knows which are current, thresholds drift, and the incident review finds the monitor that should have fired was edited by someone who left. Use this skill when you run into "Monitors as code with Terraform: resource, trio auth, explicit no-data".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.