agent's health-check loop failed: subrequest limit hit mid-poll, canary worker down

TL;DR: Cut the per-invocation fan-out below the subrequest cap. A health-check worker that fetches dozens of endpoints in one invocation exceeds the per-request subrequest budget and dies, which makes the canary report itself as down - the worst false alarm. Check a rotating subset per run, cache last-known-good results, and move polling to a scheduled worker. Refuse to start any round that plans more fetches than the budget allows.

agent's health-check loop failed: subrequest limit hit mid-poll, canary worker down

Steps

1. Count the fetches the health-check handler actually makes per invocation, including retries and redirects. Expected: the count exceeds the plan's subrequest cap - that's the bug, confirmed.
2. Cut the fan-out: check a rotating subset of endpoints per invocation, or split the checks across multiple scheduled runs. Expected: every invocation stays well under the cap.
3. Cache last-known-good results so one failed poll doesn't flip the canary to down. Expected: the canary only alerts after consecutive failures, not on a single bad round.
4. Move the polling into a scheduled (cron) worker rather than running checks per incoming request. Expected: check frequency is controlled by the schedule, not multiplied by traffic.
5. Run the checker and watch the logs for subrequest-limit errors. Expected: zero. The canary's status is trustworthy again.
6. Add the budget guard: the checker counts its planned fetches before running and refuses an over-budget round. Expected: adding endpoints later can't silently break it.

Use this when

• a health-check or canary worker throws subrequest-limit errors
• the canary reports down but the endpoints are actually fine
• "exceeded the limit of subrequests" inside polling code
• a poller that worked with few endpoints broke as endpoints were added

Not for this skill when

• the endpoints are genuinely down - the checker is right, fix the endpoints
• it's a CPU time limit, not subrequests - that's a different budget
• normal request handlers hit the limit - same fix shape, but check the fan-out there
• the failures are DNS or TLS errors - that's connectivity, not budget

Variant phrasings

• too many subrequests in a health check worker
• canary worker exceeded the subrequest limit
• health poll failed midway through its endpoint list
• poller worked until we added more endpoints

Why it happens

Every fetch inside a worker invocation counts against a fixed subrequest budget. A poller written for 10 endpoints breaks at 60 with zero code changes, and because the failure happens inside the checker itself, the canary declares itself down - exactly backwards.

Edge cases

• Redirects and retries count as subrequests too. A flaky endpoint that redirects and retries burns budget fast.
• Fetches inside waitUntil blocks count as well. Background checks aren't free.
• Free-plan caps are lower than paid. Know the plan's documented limit before sizing the fan-out.
• A rotating subset means each endpoint is checked less often. Size the rotation so the check interval still meets the alerting SLA.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pn7Gd6bPmk5tkV8mrO1qYw