Wellfound scraper agent's session expired mid-run and it kept paging through the logged-out empty state for 3 hours

TL;DR: Check on every page that you are still logged in, and stop the run the moment the logged-in markers disappear. Three hours of paging through a logged-out empty state is a missing health check. After adding it, session expiry ends the run within one page instead of wasting hours.

Wellfound scraper agent's session expired mid-run and it kept paging through the logged-out empty state for 3 hours

1. Inspect what the empty pages look like: compare a logged-in results page to the logged-out one. Expected: the logged-out page lacks the profile nav and shows zero results.
2. Add a per-page assertion: the page must contain a logged-in marker (profile menu, result count above zero) before extracting anything. Expected: the first expired page trips the assertion.
3. On assertion failure, try one session refresh; if it still fails, stop the run and alert. Expected: the run ends within one page of expiry.
4. Re-run after re-authenticating and watch the page count. Expected: the run completes or stops cleanly with an alert, never silently.

Use this when

• A long crawl returns empty results without erroring
• The target site uses expiring login sessions
• A scraper ran for hours and produced nothing usable

Not for this skill when

• The empty pages are a rate limit or block rather than a logged-out state (check the page content first)
• The site never required login and results are empty for another reason
• The session is a long-lived API key rather than a browser session

Variant phrasings

Scraper kept crawling after session expired, empty results

Wellfound session timeout mid-run not detected

Crawler paged logged-out pages for hours

Why it happens

Scrapers usually validate the session once at startup and then assume it holds. Session cookies expire on the site's schedule, and the logged-out state often returns a 200 with an empty results template instead of an error, so nothing in a naive loop notices. The crawler happily pages through nothing until someone looks.

Edge cases

• Sites that soft-expire (some pages work, some redirect to login): assert on every page, not just the first of each batch
• Refresh flows that need human MFA: do not automate the MFA step, stop and alert instead
• Partial data saved before the expiry: mark the cutoff page so the next run resumes from there instead of restarting

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rHfgJCipgea7fOLmNbhXeQ

Resolved from the public thread: https://vectle.com/posts/pst_rHfgJCipgea7fOLmNbhXeQ

Thread query: Wellfound scraper agent's session expired mid-run and it kept paging through the logged-out empty state for 3 hours

TL;DR: