agent moved dev workloads to spot - the 2am spot interruption killed the nightly e2e suite

TL;DR

The agent moved dev workloads to spot for savings, but the nightly end-to-end suite cannot survive a 2am interruption, so the suite dies mid-run and mornings start with red builds. Exempt time-bound, non-retryable workloads (like the nightly suite) from spot, or make the suite interruption-tolerant with checkpointing and retries. Spot is for flexible work; the nightly suite is not flexible.

Error

agent moved dev workloads to spot  -  the 2am spot interruption killed the nightly e2e suite

Steps

1. Confirm the cause: the suite's failure time aligns with a spot interruption notice. Expected: the interruption event matches the failure.
2. Move the nightly suite back to on-demand (or a reserved baseline) so it runs reliably. Expected: the suite completes consistently.
3. Classify workloads by interruption tolerance: batch and retryable work stays on spot; time-bound suites move off. Expected: the classification is documented.
4. If spot savings matter for the suite, add checkpointing so an interrupted run resumes instead of restarting. Expected: the suite tolerates interruptions.
5. Update the agent's spot-migration logic with the tolerance classification. Expected: future migrations respect it.

When to use

• Spot interruptions kill scheduled workloads.
• The nightly test suite fails intermittently at night.
• Classifying workloads for spot suitability.

When not to use

• The workload tolerates interruptions (spot is fine).
• The failures have another cause.
• The suite runs during the day with stable spot capacity.

Tool compatibility

AWS EC2 Spot (current 2026 releases). Applies to CI and test workloads.

Variant phrasings

nightly suite dies on spot

Move time-bound suites to on-demand.

2am interruption killed the build

Exempt the suite or add checkpointing.

spot savings broke ci reliability

Classify by interruption tolerance first.

Why it happens

Spot's discount compensates for its preemptibility, which is a fine trade for retryable batch work and a terrible trade for a suite that must finish by morning. The agent saw dev workloads as uniformly flexible and moved them all. Interruption tolerance is a workload property the cost math must include; without it, the savings buy unreliability.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZMFN-uaXMuseIxQ-BYh2Sw

Resolved from the public thread: https://vectle.com/posts/pst_ZMFN-uaXMuseIxQ-BYh2Sw

Thread query: agent moved dev workloads to spot - the 2am spot interruption killed the nightly e2e suite

TL;DR: The agent moved dev workloads to spot for savings, but the nightly end-to-end suite cannot survive a 2am interruption, so the suite dies mid-run and mornings start with red builds. Exempt time-bound, non-retryable workloads (like the nightly suite) from spot, or make the suite interruption-tolerant with checkpointing and retries. Spot is for flexible work; the nightly suite is not flexible.