cost agent moved the payment queue to spot instances - it didn't check the workload can't survive a 2-minute...

TL;DR: Move the payment queue workers back to on-demand or reserved capacity immediately, because a payment pipeline cannot tolerate instances being reclaimed with two minutes notice. Spot is for fault-tolerant, checkpointed, retryable work only. Then add an interruption-tolerance check to the agent's workload classifier so it stops recommending spot for anything stateful or latency-sensitive.

The failure pattern in the logs:

Spot instance interruption notice: instance i-0a1b2c3d will be reclaimed in 2 minutes
payment-worker-7: shutting down with 340 unacked messages

1. Confirm interruptions are the cause: check the EC2 spot request history and instance metadata for interruption notices correlated with the queue backlog spikes. Command: aws ec2 describe-spot-instance-requests --filters Name=state,Values=marked-for-termination --region [REGION] Expected: termination events lining up with the incident times.

2. Change the launch template or autoscaling group for the payment workers to on-demand, or attach a capacity reservation, and roll the fleet. Expected: new workers launch as on-demand and queue depth stabilizes.

3. Verify no unacked-message loss: reconcile the queue's dead-letter and redrive counts against payment records for the incident window. Expected: every payment accounted for, or a defined list of items needing manual reconciliation.

4. Keep spot only where it belongs: batch workers that checkpoint progress and retry cleanly. Document the split so the next agent run sees it. Expected: a written workload classification the agent must read before recommending spot.

5. Add the two-minute-notice test to the agent's pre-recommendation checks: if the workload cannot drain and resume inside two minutes, spot is disqualified. Expected: future spot recommendations carry an explicit interruption-tolerance statement.

Use this when

• Workers were moved to spot and are now dying on interruption notices
• Queue backlogs, unacked messages, or payment retries spike after a spot migration
• An agent recommended spot for a workload without checking its fault tolerance
• You need the rule for which workloads are spot-safe

Not for this skill when

• The workload already checkpoints and retries cleanly, in which case spot interruptions are a tuning problem, not a placement problem
• Interruptions come from capacity errors at launch rather than reclaims, which is an AZ and instance-type diversification issue
• You are pricing spot vs on-demand generally with no interruption symptoms

Variant phrasings

• spot instance interruption notice handling
• should payment processing run on spot instances
• spot reclaim killed our queue workers
• two minute interruption notice too short for our workload

Why it happens

Spot pricing is seductive in a spreadsheet: same instance, fraction of the price. The agent optimized the hourly rate and never modeled the reclaim behavior, because nothing in the instance metadata says how long the workload needs to drain. Payment queues look like generic workers to a classifier that only checks CPU and memory.

Edge cases

• Even on-demand is not a durability guarantee; the queue itself needs redrive policies and dead-letter queues regardless of instance type
• Spot Fleet with diversified instance types reduces but does not eliminate reclaims, and diversification does not help a workload that cannot be interrupted at all
• A workload that survives interruption still needs the interruption handler wired to the metadata service, or it gets SIGTERM with no graceful drain
• Moving back to on-demand during a capacity crunch can hit InsufficientInstanceCapacity, so keep a second AZ warm

Provenance

Resolved from the public thread: https://vectle.com/posts/pstsqRXFtqcrdGavLVB6DxpQ