Goal: get paged when the service actually breaks, not spammed by logs.
Steps:
1. Write the log filter in Logs Explorer first and confirm it matches real errors:
```
resource.type="cloud_run_revision" severity>=ERROR resource.labels.service_name="[SERVICE]"
```
2. Create a log-based metric from that filter (counter).
3. Create an alerting policy on the metric: threshold on rate over a window (e.g. > 10 errors / 5 min), with a sensible duration so one blip does not page.
4. Add a notification channel (email, Slack, PagerDuty, webhook). Test it.
Traps:
- Alerting on raw log volume instead of error rate: deploys and traffic spikes change volume. Rate or ratio alerts are stable.
- No grouping: one bad deploy can fire hundreds of alerts. Set alert grouping and auto-close so the incident is one alert, not fifty.
- Forgetting the notification channel: policies without channels evaluate silently. Verify the channel receives a test notification.
- Severity inflation: if your app logs everything as ERROR, the filter is useless. Fix log severities in the app first.
- Alert fatigue: start with a few high-signal alerts (5xx rate, crash loops). Add more only when an incident shows a gap.
Also alert on: revision not ready after deploy (failed deploys), and Pub/Sub dead-letter arrivals if you use them.
Verify: trigger a test error, confirm the metric increments, the policy fires, and the notification arrives. Then resolve and confirm auto-close.