agent's parallel restore tests crashed the sandbox and every test failed

TL;DR

Parallel restore tests crash sandboxes when the agent scales concurrency by optimism while each restore consumes real CPU, memory, storage I/O and snapshot handles. Measure per-test resource use, set concurrency from the sandbox actual bottleneck with headroom, give each test its own isolated environment or datastore slice, and queue the rest instead of launching everything at once.

agent's parallel restore tests crashed the sandbox and every test failed

Use this when

• Sandbox hosts become unresponsive or tests fail en masse when parallelism rises.
• Tests pass when run serially with the same inputs.
• Host metrics show saturation in one resource during parallel runs.

Not for this skill when

• One specific restore crashes its VM regardless of parallelism. Image problem.
• The sandbox provisioning system fails before tests start. Provisioning stage.

Steps

1. Run two tests in parallel while capturing host CPU, memory, storage latency and network use. Verify: per-test resource profile is measured
2. Identify the saturating resource and calculate a safe concurrency for the sandbox. Verify: the concurrency limit is derived from data
3. Set the agent executor to that concurrency with a queue for pending tests. Verify: extra tests wait instead of launching
4. Isolate tests so one crash cannot corrupt shared state: separate datastores, networks or snapshot namespaces per test. Verify: a forced test crash leaves neighbours unaffected
5. Run the full test set at the tuned concurrency. Verify: all tests complete without sandbox collapse

Variant phrasings

parallel tests overload sandbox concurrency cap

The cap phrasing.

restore tests crash host resource saturation

The saturation variant.

agent test parallelism isolation queue

The isolation wording.

Compatibility: Agent test runners executing restore verifications in parallel on shared VMware, Hyper-V or cloud sandboxes.

Why it happens

Restore is one of the heaviest operations in IT: full reads, decompression, VM boot storms. A sandbox sized for a few restores behaves nothing like its idle specifications suggest, and agent frameworks default to parallelism that assumes cheap tasks. The crash is the sandbox telling the truth about physics; capacity-derived concurrency is the fix.

Edge cases / pitfalls

• Boot storms are spikier than steady restore load. Stagger VM power-ons even within the concurrency cap.
• A crashed sandbox can leave half-restored VMs registered. Cleanup must be part of the test harness, not manual.
• If business pressure demands more parallelism, the honest answer is more sandbox, not higher caps.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hsO8kTbC1qzV0It5dbOStQ