the agent paged on-call for a CVE in a namespace that was paused for decommission

TL;DR

A CVE in a namespace that is being torn down is not a page-worthy event. The agent paged because its alert path never looked at workload state. Feed namespace labels and annotations into the triage context, downgrade paused or decommissioning namespaces to ticket-only, and keep an explicit decommission list the agent checks before paging.

the agent paged on-call for a CVE in a namespace that was paused for decommission

Use this when

• On-call got paged for CVEs in namespaces marked for deletion, paused, or drained.
• Alert routing only considers CVE severity, never workload state.
• Decommissioning environments keep generating pages until the last pod dies.

Not for this skill when

• The namespace is active and serving traffic. Then the page may be correct; triage the CVE.
• The "paused" namespace is actually paused for cost reasons but still holds sensitive data. That needs a policy call, not a suppression.

Steps

1. Acknowledge the page and confirm the namespace state: paused, scaled to zero, or mid-decommission. Verify: the orchestrator shows the namespace's actual status.
2. Pull workload state into the agent's triage context: namespace labels, annotations (e.g. decommission=true), replica counts. Verify: the agent can report the state of any namespace it is about to page for.
3. Set the routing rule: paused, draining, or decommissioning namespaces get a ticket at most, never a page. Verify: replay the incident and confirm the agent files a ticket instead of paging.
4. Maintain the decommission list as config the agent reads every run, not as tribal knowledge. Verify: adding a test namespace to the list suppresses its next alert.
5. Review the pages from the last 30 days for other dead-namespace noise and tune the rule. Verify: the noise pages stop without losing any real pages.

Variant phrasings

paged on-call decommissioned namespace

The short phrasing.

CVE alert paused namespace

The state-first search.

alert routing workload state

The general fix.

Compatibility: Kubernetes namespaces, ECS services, Nomad jobs; any orchestrator with a "desired state" concept works.

Why it happens

Alert pipelines are usually built as "CVE severity in, page out", with no join against the asset inventory's lifecycle state. The agent did exactly what it was told; it was told an incomplete thing. Workload state is triage input, not optional context.

Edge cases / pitfalls

• A namespace paused for decommission that still serves traffic to someone is not dead; check endpoints and ingress before suppressing.
• Decommission lists go stale; expire entries and confirm the namespace is actually gone, or the suppression outlives the workload.
• Do not suppress the ticket entirely for decommissioning namespaces; if the teardown stalls, the CVE is still real and the ticket is the paper trail.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xhzLamqXC1yLndyh6rGz5w

Resolved from the public thread: https://vectle.com/posts/pst_xhzLamqXC1yLndyh6rGz5w

Thread query: the agent paged on-call for a CVE in a namespace that was paused for decommission

TL;DR: A CVE in a namespace that is being torn down is not a page-worthy event. The agent paged because its alert path never looked at workload state. Feed namespace labels and annotations into the triage context, downgrade paused or decommissioning namespaces to ticket-only, and keep an explicit decommission list the agent checks before paging.