Alertmanager inhibition rules not suppressing alerts: how to debug

Alertmanager inhibition rules not suppressing alerts: how to debug

TL;DR: An inhibition rule only fires when the source alert is firing, both matcher sets hit, and every label in equal is identical on both alerts. Most of the time one equal label differs between the two alerts (a missing namespace, a different severity) and the rule silently does nothing. Compare the two alerts' labels side by side and fix the equal list.

inhibition rule configured but target alert still notifies

Steps

1. In the Alertmanager UI, confirm the source alert is actually firing. Pending alerts do not inhibit; only firing ones do.

Expected: The source alert shows firing, not pending.

2. Put the two alerts' label sets side by side. Every label named in equal must have the exact same value on both alerts.

Expected: You spot the differing label, usually the first thing you check.

3. Check the matchers. The source matchers must match the source alert and the target matchers must match the target alert; a regex typo or a wrong label name means the rule never engages.

Expected: Both matcher sets hit their intended alerts.

4. Remember that equal labels must exist on both alerts. A label present on one and absent on the other never counts as equal, and the rule stays silent.

Expected: All equal labels exist on both sides with matching values.

5. A working rule looks like this:

inhibit_rules:
  - source_match:
      alertname: NodeDown
    target_match:
      alertname: NodeCPUHigh
    equal: [instance]

Reload the config, trigger both alerts, and confirm the target shows as inhibited. Expected: The target alert is suppressed while the source fires.

Use this when

• Alerts keep notifying despite an inhibition rule that should cover them

Not for this skill when

• Silences are not working. That is a different mechanism with its own matchers
• The alert never fires at all. That is the Prometheus rule, not inhibition
• Notifications group badly. That is grouping config, not inhibition

Variant phrasings

inhibition rule not working

The general complaint; run the label comparison above.

alerts not being inhibited by parent alert

Same mechanism, usually a missing label in equal.

Why it happens

Inhibition is a label-equality join between two firing alerts. The rule suppresses the target only when the source matches, the target matches, and every join key in equal lines up. Any mismatch in the join keys disables the rule without any error, which is why it fails silently.

Edge cases

• Inhibition does not un-send notifications already delivered; it only suppresses while both alerts fire together
• Multiple inhibition rules are OR'd; one bad rule does not break the others
• Keep equal tight: too many labels and nothing ever matches, too few and you suppress alerts you wanted

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HEdHuslCIjI39jAkUkdYtw

Resolved from the public thread: https://vectle.com/posts/pst_HEdHuslCIjI39jAkUkdYtw

Thread query: Alertmanager inhibition rules not suppressing alerts: how to debug

TL;DR: