agent failed to enforce canary gate because status webhook never arrived

TL;DR

A gate that depends on one webhook delivery is a single point of failure: webhooks get lost, endpoints rotate, signatures fail. Make the gate decision from polling the source of truth (the canary hosts health and patch state) with the webhook as a mere accelerator, and define the timeout behavior explicitly: on gate timeout, halt and escalate, never auto-proceed.

agent failed to enforce canary gate because status webhook never arrived

Use this when

• The wave sits at the canary gate for hours although canary hosts are healthy.
• Webhook delivery logs show failures, retries, or signature errors.
• Someone manually advanced the wave, defeating the gate purpose.

Not for this skill when

• Canary hosts failed health checks and the gate correctly held. That is the gate working; fix the canary outcome, not the gate.
• The webhook arrives but the payload is misread. That is a parser problem; see the parsing skills in this vertical.

Steps

1. Identify the gate data source of truth (canary host patch state plus health checks) independent of the webhook. Verify: you can evaluate the gate manually from that data alone
2. Implement polling evaluation at a sane interval (for example every 2 to 5 minutes) as the gate mechanism, with webhook receipt only triggering an early evaluation. Verify: a test gate evaluates on poll without any webhook
3. Set an explicit gate timeout with halt-and-escalate semantics and an alert to the wave owner. Verify: a simulated never-ready canary produces an escalation, not a silent proceed
4. Log gate decisions with the evidence snapshot used (host states, health results, timestamps). Verify: the wave record shows why the gate passed or held
5. Fix the webhook as a secondary matter (endpoint, secret rotation, retries) and monitor its delivery rate. Verify: webhook delivery success is visible but the gate no longer depends on it

Variant phrasings

canary gate webhook never arrived polling fallback

The fix-pattern phrasing.

gate timeout halt escalate never auto proceed patch

The safety-semantics phrasing.

webhook single point of failure deployment gate

The design-lesson phrasing.

Compatibility: Compatibility: Any gated rollout system (patch waves, deployment pipelines) with a health data source queryable by API.

Why it happens

Webhooks are at-most-once notifications in most real systems: a missed delivery, a rotated signing secret, or a brief endpoint outage silently removes the signal. A safety gate, whose entire job is to stop bad rollouts, must be built on queryable state, with notifications as an optimization. The second design error is ambiguous timeout: if waiting long enough eventually proceeds, the gate is decorative.

Edge cases / pitfalls

• Test the gate by breaking it: kill the webhook in staging and prove the wave still gates correctly.
• Canary health must include the patched service real function, not just host up, or the gate passes on a broken canary.
• Record gate bypasses as incidents even when the outcome was fine; the bypass, not the outcome, is the risk.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_J9X3jT8XsNWkZbVUkqtUYA