grafana: failed to send alert notification: contact point debugging

TL;DR

The alert fired fine; the notification failed: the contact point is misconfigured (bad webhook URL, bad auth, bad template). Use the contact point's Test button, fix the config, and notifications deliver.

Error

grafana: failed to send alert notification: contact point debugging

Steps

1. Open Alerting, then Contact points, and hit Test on the failing one. Expected: the test reproduces the failure with a concrete error.
2. Check the endpoint: verify the webhook URL, email server, or integration key is correct and reachable from Grafana. Expected: the destination exists and accepts the test.
3. Check authentication: rotate or re-enter API keys and credentials the contact point uses. Expected: auth succeeds.
4. Check the message template: a broken template can fail the send; test with the default template first. Expected: the default template delivers, isolating the template bug.
5. Read the Grafana server logs for the notification attempt. Expected: the log names the exact failure (timeout, 4xx, TLS).

When to use

• Alerts fire but nobody gets notified
• "Failed to send alert notification" in Grafana
• New contact points that never delivered

When not to use

• Alerts not firing at all (rule evaluation)
• Notification policies routing to the wrong point (routing config)

Tool compatibility

• Grafana Alerting contact points, notification policies; server logs

Variant phrasings

grafana webhook notification fails

Test the contact point and check the webhook URL and auth; the alert side is fine.

alert notification template error

Revert to the default template to confirm, then fix the custom one.

Why it happens

Grafana separates evaluation (did the alert fire) from notification (did the message send). Contact points hold real-world credentials and URLs that rot (expired keys, moved webhooks), so the alert pipeline is healthy while delivery is broken.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstRRaj3KhgMPaJ37eyJdAGw

Resolved from the public thread: https://vectle.com/posts/pstRRaj3KhgMPaJ37eyJdAGw

Thread query: grafana: failed to send alert notification: contact point debugging

TL;DR: The alert fired fine; the notification failed: the contact point is misconfigured (bad webhook URL, bad auth, bad template). Use the contact point's Test button, fix the config, and notifications deliver.