InvalidVolume.NotFound" when the cleanup agent tried to delete a volume another run already removed

TL;DR

Two cleanup runs raced: the first deleted the volume, and the second's delete call found nothing, which is actually the desired end state. Catch InvalidVolume.NotFound in the delete path and treat it as success (log it as already-removed), and serialize or deduplicate cleanup runs so they stop racing. Not-found on delete means the goal is already achieved.

Error

"InvalidVolume.NotFound" when the cleanup agent tried to delete a volume another run already removed

Steps

1. Confirm the race: check the agent logs for two runs targeting the same volume ID. Expected: both runs list the volume, one delete succeeded, the other hit NotFound.
2. In the agent's delete function, catch InvalidVolume.NotFound and log it as already-removed instead of failing. Expected: the second run completes cleanly.
3. Add run coordination: a lock, a queue, or a schedule that prevents overlapping cleanup runs on the same scope. Expected: concurrent runs no longer target the same volumes.
4. Make the whole delete path idempotent: describe first, delete only if present, tolerate NotFound. Expected: reruns are safe no matter the starting state.
5. Re-run the cleanup and verify the log shows clean already-removed entries. Expected: no failures, no duplicate work.

When to use

• A cleanup agent fails with InvalidVolume.NotFound on volumes another run deleted.
• Overlapping scheduled runs race on the same inventory.
• Making delete operations idempotent.

When not to use

• The volume exists but deletion fails with VolumeInUse (a state problem).
• The volume was never in the inventory (a targeting bug).
• The goal is to prevent deletion rather than tolerate it.

Tool compatibility

AWS EC2 EBS (current 2026 releases). Applies to cleanup agents in any language.

Variant phrasings

"the volume does not exist" on delete

Same race in plain wording. Treat as success.

two cleanup schedules deleting the same volumes

Serialize the runs or partition the inventory.

delete failed because it was already gone

Idempotent delete: NotFound is the goal state.

Why it happens

Cleanup agents enumerate doomed resources and then act, but the enumeration goes stale the moment another actor (a second run, a human, an autoscaler) deletes something first. Delete APIs are strict: deleting a nonexistent volume is an error, not a no-op. Idempotent handling converts the race from a failure into a benign log line, and run coordination removes the race at the source.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Izax-FX7PR-VksbcP2bOhA