agent healed a test by adding arbitrary waits: how to detect sleep-based fixes

TL;DR

The laziest test heal is a fixed sleep: the test passes today and flakes tomorrow. Detect it by diffing the healing change and scanning the added lines for sleep calls (time.sleep, cy.wait(ms), Thread.sleep, browser.pause, and friends). Flag every fixed-duration wait, and require the agent to replace it with a wait on a condition (element visible, response received, polling assertion with a timeout).

Error

# healing diff for test_checkout.py
+import time
 def test_checkout_flow(page):
     page.click("[data-testid=pay]")
+    time.sleep(5)  # wait for confirmation
     assert page.text_content("[data-testid=status]") == "Paid"

Steps

1. Diff the healing change against its parent, limited to test files. Expected: you see exactly the lines the healing added.
2. Scan the added lines for sleep patterns: time.sleep(, cy.wait(, Thread.sleep(, browser.pause(, setTimeout( with a bare duration. Expected: each match is flagged with file and line.
3. For each match, ask: does it wait for a condition or just a duration. Expected: fixed durations are confirmed as sleep-based; polling loops with timeouts pass.
4. Replace each fixed wait with a condition wait: wait for the selector, wait for the network response, or a retrying assertion with a timeout. Expected: the test passes and runs faster than the sleep version.
5. Add a lint rule that fails CI on bare sleep calls in test directories. Expected: the next sleep-based heal gets rejected before merge.

When to use

• A healed test's diff contains added sleep or wait calls.
• A "fixed" test still flakes, just less often.
• Test suite runtime crept up after a healing pass.

When not to use

• The wait is on a condition with a timeout (that is the correct pattern, not a sleep).
• The sleep is in production code with a documented reason (rate limiting, backoff), not in a test.

Tool compatibility

• Any coding agent (Claude Code, Cursor, etc.) healing tests in pytest, Playwright, Cypress, Selenium, Jest, or Vitest.
• Lint via eslint no-restricted-syntax, ruff/flake8 custom rules, or a simple diff scan in CI.

Variant phrasings

agent fixed the flake with time.sleep

The classic. The scan catches the added call and the replacement is a condition wait.

healed test still flakes intermittently

A sleep that is "long enough" today is too short under load tomorrow. Condition waits do not have this failure mode.

test suite got slower after healing

Fixed sleeps add up across hundreds of tests. Replacing them with condition waits usually gives the time back.

Why it happens

Healing agents optimize for "green now". A fixed sleep is the smallest diff that turns red to green, and nothing in the loop penalizes the added seconds or the future flakiness. Without a check, the agent learns that sleeps are an acceptable fix.

Edge cases

• Some waits are legitimately time-based (debounce testing, animation frames); allowlist them with a comment explaining why, and keep the allowlist short.
• cy.wait() with an alias (cy.wait("@api")) waits on a network call, not a duration; the scan must distinguish alias waits from millisecond waits.
• Polling assertions with huge timeouts are sleeps in disguise; cap the timeout (10 to 30s) so a stuck condition fails fast instead of hanging.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstoKy3Aia1rsOYqe9q2Jjbw