agent timed out waiting on its own canary worker: CPU time limit hit mid load test, canary failed

TL;DR: Move load generation off the edge. A canary worker has a fixed CPU budget per invocation, and using it to generate load spends that budget on the generator instead of the check - so the canary CPU-starves, times out, and the agent misreads it as the target being down. Run the load tool from your own machine or CI, keep the canary handler light, and base the agent's wait timeout on the external tool's completion.

agent timed out waiting on its own canary worker: CPU time limit hit mid load test, canary failed

Steps

1. Confirm the failure mode: check the worker logs for CPU time limit errors during the test window. Expected: the errors line up with the load test, proving the generator starved itself.
2. Move load generation off the edge: run the load tool from your own machine or a CI runner, pointed at the canary URL. Expected: the canary only runs its light check logic per request.
3. Slim the canary handler: no parsing huge bodies, no crypto loops, no busy work in the hot path. Expected: handler CPU stays in the low milliseconds per invocation.
4. Re-run the load test externally and watch the canary's error rate. Expected: the canary stays green; any failures now indicate real capacity problems, not self-inflicted ones.
5. Base the agent's wait timeout on the external load tool's completion, not on the worker responding mid-test. Expected: no more timeouts caused by a CPU-starved canary.
6. Document the canary's CPU budget and add a pre-deploy check that the handler does no heavy compute. Expected: load-test-shaped traffic never runs inside the canary again.

Use this when

• a canary worker dies with CPU time limit errors during a synthetic load test
• the agent times out waiting on a canary that's also generating load
• "CPU time limit hit mid load test"
• load-test results show the canary failing before the target does

Not for this skill when

• production handlers hit CPU limits under real traffic - optimize the handler
• it's a subrequest or memory limit, not CPU - different budget, different fix
• the target genuinely can't take the load - the canary is right
• the load test runs externally already and the canary still fails - investigate the target

Variant phrasings

• worker exceeded CPU time during a synthetic load test
• canary went down during load testing on CPU limits
• load testing from inside a worker process
• agent's own load generator starved the canary

Why it happens

A worker invocation gets a fixed CPU budget, and generating load is CPU-hungry work. Running the generator inside the canary means the check logic competes with the load logic for the same budget and loses - the canary times out and reports the target as down when the target was never the problem.

Edge cases

• Waiting on fetch doesn't burn CPU; computing does. I/O-heavy checks are cheap, parse-heavy checks aren't.
• Paid plans raise the CPU limit but don't make in-worker load generation sane. The architecture is wrong at any budget.
• Burst vs sustained limits differ. A short spike passes where a long test fails - size the test to the sustained budget.
• If the canary must do crypto (signed checks), keep it minimal and cache verification results.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KKmSSOKibEGsKszHS-oMug