## TL;DR

The agent moved dev workloads to spot for savings, but the nightly end-to-end suite cannot survive a 2am interruption, so the suite dies mid-run and mornings start with red builds. Exempt time-bound, non-retryable workloads (like the nightly suite) from spot, or make the suite interruption-tolerant with checkpointing and retries. Spot is for flexible work; the nightly suite is not flexible.

## Error

```text
agent moved dev workloads to spot  -  the 2am spot interruption killed the nightly e2e suite
```

## Steps

1. Confirm the cause: the suite's failure time aligns with a spot interruption notice. Expected: the interruption event matches the failure.
2. Move the nightly suite back to on-demand (or a reserved baseline) so it runs reliably. Expected: the suite completes consistently.
3. Classify workloads by interruption tolerance: batch and retryable work stays on spot; time-bound suites move off. Expected: the classification is documented.
4. If spot savings matter for the suite, add checkpointing so an interrupted run resumes instead of restarting. Expected: the suite tolerates interruptions.
5. Update the agent's spot-migration logic with the tolerance classification. Expected: future migrations respect it.

## When to use

- Spot interruptions kill scheduled workloads.
- The nightly test suite fails intermittently at night.
- Classifying workloads for spot suitability.

## When not to use

- The workload tolerates interruptions (spot is fine).
- The failures have another cause.
- The suite runs during the day with stable spot capacity.

## Tool compatibility

AWS EC2 Spot (current 2026 releases). Applies to CI and test workloads.

### Variant phrasings

### nightly suite dies on spot

Move time-bound suites to on-demand.

### 2am interruption killed the build

Exempt the suite or add checkpointing.

### spot savings broke ci reliability

Classify by interruption tolerance first.

## Why it happens

Spot's discount compensates for its preemptibility, which is a fine trade for retryable batch work and a terrible trade for a suite that must finish by morning. The agent saw dev workloads as uniformly flexible and moved them all. Interruption tolerance is a workload property the cost math must include; without it, the savings buy unreliability.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZMFN-uaXMuseIxQ-BYh2Sw