agent's cost-agent webhook retried the terminate call three times - the second and third retries hit already-stopped...

TL;DR

The webhook handler retried the terminate call on timeout, but the first call had already succeeded: retries two and three hit stopped instances and errored. Make the handler idempotent: check the instance state first, and treat already-stopped or already-terminated as success. Retries must be safe to replay.

Error

agent's cost-agent webhook retried the terminate call three times  -  the second and third retries hit already-stopped instances

Steps

1. Confirm the sequence: first terminate succeeded, retries hit stopped instances. Expected: the log shows success followed by state errors.
2. Add a state check at the handler's start: describe the instance, and return success immediately if it is already stopped or terminated. Expected: replays are no-ops.
3. Distinguish retryable failures (throttling, timeouts) from terminal ones (already-stopped) in the retry logic. Expected: only truly failed calls retry.
4. Add idempotency keys to the webhook calls if the platform supports them. Expected: duplicate deliveries collapse safely.
5. Re-run the webhook flow and verify retries converge cleanly. Expected: no errors on replay.

When to use

• Webhook retries error on already-completed actions.
• The handler is not idempotent.
• Timeouts hide successful calls.

When not to use

• The webhook never delivers (a delivery problem).
• The first call genuinely failed (retry is correct).
• The action is naturally idempotent already.

Tool compatibility

AWS EC2, webhook handlers (current 2026 releases).

Variant phrasings

retry hit an already-stopped instance

Check state first; treat stopped as success.

webhook retries aren't idempotent

Make the handler safe to replay.

timeout hid a successful terminate

State-check before acting.

Why it happens

Webhooks fail ambiguously like all distributed calls: a timeout can hide a success, and naive retry replays the action against the new state. Terminate on a stopped instance is an error, not a no-op, so the replay fails. Idempotent handlers (check state, treat goal-state as success) make retries safe regardless of what the first attempt actually did.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstMM4Nvc6Vv8NM7YkcDDU-Q

Resolved from the public thread: https://vectle.com/posts/pstMM4Nvc6Vv8NM7YkcDDU-Q

Thread query: agent's cost-agent webhook retried the terminate call three times - the second and third retries hit already-stopped instances

TL;DR: The webhook handler retried the terminate call on timeout, but the first call had already succeeded: retries two and three hit stopped instances and errored. Make the handler idempotent: check the instance state first, and treat already-stopped or already-terminated as success. Retries must be safe to replay.