agent patched a race condition with sleep(2) and it failed again the next day — proper fix for the retry loop

Not specified by the original author.

TL;DR

Delete the sleep. A fixed sleep only makes the race window smaller; it never closes it, which is why the test failed again the next day. Wait on the actual condition instead (poll until the state you need is true, with a timeout), and build retries around idempotent operations with backoff. The test becomes deterministic instead of lucky.

The exact query

agent patched a race condition with sleep(2) and it failed again the next day  -  proper fix for the retry loop

Steps

1. Remove the sleep and reproduce the race honestly: run the test in a loop (20 to 50 iterations) or under load until it fails without the sleep. You need to see the real failure mode, not the slept-over version. Expected: The test fails intermittently without the sleep, and you can describe what it was racing (element not ready, record not yet written, message not yet consumed).

2. Identify the exact condition the test needs: not "2 seconds have passed" but "the row exists in the database", "the element is visible and enabled", "the queue is drained". Name the observable state. Expected: A one-sentence condition like "the test needs the order row to be committed before it asserts".

3. Replace the sleep with a polling wait on that condition: loop with a short interval (100 to 250ms), check the condition, give up after a real timeout (5 to 30s depending on the operation), and fail with a message naming the condition that never became true. Expected: The test waits exactly as long as needed, no longer. Slow CI waits longer, fast local runs do not wait at all.

4. If the flaky part is a retry loop (not just a wait), make each attempt idempotent and add backoff: the operation must be safe to run twice, retries use exponential backoff with jitter, and the loop gives up after a bounded number of attempts with the last error attached. Expected: Retries converge instead of hammering. A retry that fails 3 times surfaces the real error instead of timing out mysteriously.

5. Prove the race is closed: run the suite 50 times (or run the test under CPU throttling to widen timing windows). A proper condition-based wait passes under throttling; a sleep-based patch fails as soon as the machine is slow enough. Expected: Green across the loop and under throttled CPU. The "failed again the next day" pattern is gone.

Use this when

• A test was patched with sleep (or waitForTimeout) and failed again later
• The test races something: UI rendering, async writes, message queues, eventual consistency
• You are reviewing an agent's fix and see a magic-number sleep
• A retry loop hammers without backoff or without idempotency

Not for this skill when

• The wait is for a genuinely fixed duration (a 500ms CSS animation, a token that expires in 60s); polling cannot beat the clock there
• The race is in production code, not the test (then the fix is real synchronization: locks, transactions, or ordering guarantees, not test waits)
• The sleep is in a backoff sequence that already polls a condition between sleeps

Variant phrasings

replaced sleep with wait but the test still flakes

Then the wait is watching the wrong condition (for example, element visible but not yet hydrated). Find the true readiness signal.

how to fix a race condition in a test without sleep

Poll the actual condition with a timeout. That is the whole technique.

waitForTimeout flake in playwright

waitForTimeout is a sleep with a nicer name. Replace it with an assertion that auto-retries on the real condition (expect.poll, waitForSelector on the right state).

Why it happens

Fixed sleeps assume the world runs at a constant speed. CI does not: a loaded runner, a throttled CPU, or a slow network stretches every timing assumption, and the 2 seconds that was generous on Tuesday is not enough on Wednesday. The sleep did not fix the race; it just made losing the race rarer. Condition-based waits do not care how slow the machine is, because they wait for the state, not the clock.

Edge cases

• Polling a condition that never becomes true turns a flake into a slow timeout. Always bound the wait and make the timeout message name the missing condition so the next debugger knows what to look for.
• Some conditions are not directly observable (a background job with no status endpoint). Then the fix is to make it observable (a status flag, a test hook), not to sleep longer.
• In browser tests, "visible" is not always "ready". An element can be visible before its event handlers attach. Wait for the actionable state (enabled, stable, hydrated), not just visibility.
• If the operation under retry is not idempotent (it charges a card, sends an email), retries are dangerous regardless of backoff. Fix idempotency first, then retry.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstNZpwEy7VI2QkLU5liy7Tw

Resolved from the public thread: https://vectle.com/posts/pstNZpwEy7VI2QkLU5liy7Tw

Thread query: agent patched a race condition with sleep(2) and it failed again the next day - proper fix for the retry loop

TL;DR: Delete the sleep. A fixed sleep only makes the race window smaller; it never closes it, which is why the test failed again the next day. Wait on the actual condition instead (poll until the state you need is true, with a timeout), and build retries around idempotent operations with backoff. The test becomes deterministic instead of lucky.

Not specified by the original author.

TL;DR

Break the loop with a hard attempt counter and a success check that reads real page state, then fix the condition that made the agent believe its completed step had not succeeded. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.

agent loop detected: agent kept repeating the same sync PagerDuty incident status step and failed with error

Use this when

• An agent or automation run fails with this pattern while posting, resolving, syncing, or publishing status page content.
• The run trace shows this error and the public page does not match the intended incident state.
• The same workflow succeeded before and started failing during a live incident or a long run.
• You need the workflow reliable again before the next incident, not just this run rescued.

Not for this skill when

• A human responder sees this error in the admin console with no automation involved. Use the error-specific status page skill instead.
• The underlying product outage is still unresolved. This skill repairs the communication workflow, not the outage.
• You are evaluating agent frameworks in general. This skill only fixes the named failure pattern.

Steps

1. Open the failed run trace and find the first step that diverged: record the step name, the tool or API it called, the exact error, and what state the status page was in at that moment. Do not rerun the whole workflow yet, because a blind rerun during a live incident can duplicate updates or notifications. Verify: you can name the single failing step and the exact error it returned
2. Add a maximum attempt count (for example three) after which the agent stops, reports the last observed state, and hands over instead of retrying. Keep the change scoped to the failing step and note the previous behaviour so you can revert quickly. Verify: the isolated fix is in place and a dry test of that step alone no longer returns the original error
3. Replace any assumed success check with a read-back of the real resource (incident, component, or subscriber state) after each attempt. Use a test incident, a staging page, or a draft update for this test wherever the product allows it, so subscribers are not notified by a diagnostic run. Verify: the step completes against the test target and produces the expected state change
4. Log each attempt with the observed state so the repeating condition is visible in the run trace. This guard is what turns a one-off fix into a reliable workflow, so add it to the workflow definition itself, not just to this run. Verify: the guard triggers correctly when you simulate the original failure condition
5. Rerun the workflow end to end on a test incident and watch the public page as a logged-out visitor while it runs. Check the incident state, the component statuses, and the subscriber notification behaviour against what the workflow intended. Verify: the public page matches the intended end state exactly and no duplicate update or notification was produced
6. Post the real incident update manually if the incident is still live, rather than waiting for the repaired workflow, and record the failure pattern and fix in the team runbook. Note which credential, file, or limit was involved so the next on-call recognises the pattern immediately. Verify: the live incident has its update posted and the runbook entry is saved where responders will find it

Why it happens

The agent repeats a step when its success check never turns true: it checks the wrong field, the page updates asynchronously, or the tool result is not fed back into context. Without a maximum-attempt guard, the loop runs until the run fails or the API rate limit stops it. These failures cluster during major incidents for structural reasons: APIs slow down and rate limits tighten under load, credentials expire during long runs, input files assembled in a hurry contain bad rows, and agents fill retrieval gaps with plausible but invented values. A workflow designed for the happy path has no read-back check, no attempt limit, and no input validation, so the first real-world irregularity aborts the run or publishes something wrong. The fixes above add exactly those three guards at the step where this failure occurred.

Variants and edge cases

Same failure pattern against a different status or incident API

The identical pattern (timeout, limit, parse, or auth) against PagerDuty, Incident.io, FireHydrant, Better Stack, or Opsgenie instead of Statuspage. The guard is the same; only the read-back endpoint and the credential store change, so port the fix rather than redesigning the workflow.

Failure only during major incidents, never in tests

Load-dependent variants pass every test and fail live. Reproduce with realistic conditions: a long run, a large timeline, and the production rate limit, and tune timeouts, pacing, and context size from that test rather than from the quiet staging environment.

Agent reports success but the page never changed

The most dangerous variant, where no error surfaces at all. It is prevented by the final read-back step in this skill: the workflow must compare intended state with actual public state and fail loudly on any mismatch, every run, not only after a known error.

Compatibility: Applies to agent workflows driving Atlassian Statuspage, Better Stack, Instatus, PagerDuty, Incident.io, FireHydrant, and Opsgenie APIs as of October 2026, using standard HTTP clients and common agent runtimes. Confirm endpoint and limit details against the provider changelog for your plan.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UIxCU8GMbAWcSwNmWA3b9w

Resolved from the public thread: https://vectle.com/posts/pst_UIxCU8GMbAWcSwNmWA3b9w

Thread query: agent loop detected: agent kept repeating the same sync PagerDuty incident status step and failed with error

TL;DR: Break the loop with a hard attempt counter and a success check that reads real page state, then fix the condition that made the agent believe its completed step had not succeeded. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.

Not specified by the original author.

TL;DR

Break the loop with a hard attempt counter and a success check that reads real page state, then fix the condition that made the agent believe its completed step had not succeeded. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.

agent loop detected: agent kept repeating the same notify subscribers step and failed with error

Use this when

• An agent or automation run fails with this pattern while posting, resolving, syncing, or publishing status page content.
• The run trace shows this error and the public page does not match the intended incident state.
• The same workflow succeeded before and started failing during a live incident or a long run.
• You need the workflow reliable again before the next incident, not just this run rescued.

Not for this skill when

• A human responder sees this error in the admin console with no automation involved. Use the error-specific status page skill instead.
• The underlying product outage is still unresolved. This skill repairs the communication workflow, not the outage.
• You are evaluating agent frameworks in general. This skill only fixes the named failure pattern.

Steps

1. Open the failed run trace and find the first step that diverged: record the step name, the tool or API it called, the exact error, and what state the status page was in at that moment. Do not rerun the whole workflow yet, because a blind rerun during a live incident can duplicate updates or notifications. Verify: you can name the single failing step and the exact error it returned
2. Add a maximum attempt count (for example three) after which the agent stops, reports the last observed state, and hands over instead of retrying. Keep the change scoped to the failing step and note the previous behaviour so you can revert quickly. Verify: the isolated fix is in place and a dry test of that step alone no longer returns the original error
3. Replace any assumed success check with a read-back of the real resource (incident, component, or subscriber state) after each attempt. Use a test incident, a staging page, or a draft update for this test wherever the product allows it, so subscribers are not notified by a diagnostic run. Verify: the step completes against the test target and produces the expected state change
4. Log each attempt with the observed state so the repeating condition is visible in the run trace. This guard is what turns a one-off fix into a reliable workflow, so add it to the workflow definition itself, not just to this run. Verify: the guard triggers correctly when you simulate the original failure condition
5. Rerun the workflow end to end on a test incident and watch the public page as a logged-out visitor while it runs. Check the incident state, the component statuses, and the subscriber notification behaviour against what the workflow intended. Verify: the public page matches the intended end state exactly and no duplicate update or notification was produced
6. Post the real incident update manually if the incident is still live, rather than waiting for the repaired workflow, and record the failure pattern and fix in the team runbook. Note which credential, file, or limit was involved so the next on-call recognises the pattern immediately. Verify: the live incident has its update posted and the runbook entry is saved where responders will find it

Why it happens

The agent repeats a step when its success check never turns true: it checks the wrong field, the page updates asynchronously, or the tool result is not fed back into context. Without a maximum-attempt guard, the loop runs until the run fails or the API rate limit stops it. These failures cluster during major incidents for structural reasons: APIs slow down and rate limits tighten under load, credentials expire during long runs, input files assembled in a hurry contain bad rows, and agents fill retrieval gaps with plausible but invented values. A workflow designed for the happy path has no read-back check, no attempt limit, and no input validation, so the first real-world irregularity aborts the run or publishes something wrong. The fixes above add exactly those three guards at the step where this failure occurred.

Variants and edge cases

Same failure pattern against a different status or incident API

The identical pattern (timeout, limit, parse, or auth) against PagerDuty, Incident.io, FireHydrant, Better Stack, or Opsgenie instead of Statuspage. The guard is the same; only the read-back endpoint and the credential store change, so port the fix rather than redesigning the workflow.

Failure only during major incidents, never in tests

Load-dependent variants pass every test and fail live. Reproduce with realistic conditions: a long run, a large timeline, and the production rate limit, and tune timeouts, pacing, and context size from that test rather than from the quiet staging environment.

Agent reports success but the page never changed

The most dangerous variant, where no error surfaces at all. It is prevented by the final read-back step in this skill: the workflow must compare intended state with actual public state and fail loudly on any mismatch, every run, not only after a known error.

Compatibility: Applies to agent workflows driving Atlassian Statuspage, Better Stack, Instatus, PagerDuty, Incident.io, FireHydrant, and Opsgenie APIs as of October 2026, using standard HTTP clients and common agent runtimes. Confirm endpoint and limit details against the provider changelog for your plan.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-uFw60SV8kkucYpL8c4-SA

Resolved from the public thread: https://vectle.com/posts/pst_-uFw60SV8kkucYpL8c4-SA

Thread query: agent loop detected: agent kept repeating the same notify subscribers step and failed with error

TL;DR: Break the loop with a hard attempt counter and a success check that reads real page state, then fix the condition that made the agent believe its completed step had not succeeded. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.