Wellfound scraper agent's session expired mid-run and it kept paging through the logged-out empty state for 3 hours
Fixes a Wellfound scraper that kept paging through a logged-out empty state for hours after its session expired. Use when a long crawl returns nothing but never errors. Key trigger: pages with zero results and no logged-in markers long after the run started.
TL;DR: Check on every page that you are still logged in, and stop the run the moment the logged-in markers disappear. Three hours of paging through a logged-out empty state is a missing health check. After adding it, session expiry ends the run within one page instead of wasting hours.
Wellfound scraper agent's session expired mid-run and it kept paging through the logged-out empty state for 3 hours- Inspect what the empty pages look like: compare a logged-in results page to the logged-out one. Expected: the logged-out page lacks the profile nav and shows zero results.
- Add a per-page assertion: the page must contain a logged-in marker (profile menu, result count above zero) before extracting anything. Expected: the first expired page trips the assertion.
- On assertion failure, try one session refresh; if it still fails, stop the run and alert. Expected: the run ends within one page of expiry.
- Re-run after re-authenticating and watch the page count. Expected: the run completes or stops cleanly with an alert, never silently.
Use this when
- A long crawl returns empty results without erroring
- The target site uses expiring login sessions
- A scraper ran for hours and produced nothing usable
Not for this skill when
- The empty pages are a rate limit or block rather than a logged-out state (check the page content first)
- The site never required login and results are empty for another reason
- The session is a long-lived API key rather than a browser session
Variant phrasings
Scraper kept crawling after session expired, empty results
Wellfound session timeout mid-run not detected
Crawler paged logged-out pages for hours
Why it happens
Scrapers usually validate the session once at startup and then assume it holds. Session cookies expire on the site's schedule, and the logged-out state often returns a 200 with an empty results template instead of an error, so nothing in a naive loop notices. The crawler happily pages through nothing until someone looks.
Edge cases
- Sites that soft-expire (some pages work, some redirect to login): assert on every page, not just the first of each batch
- Refresh flows that need human MFA: do not automate the MFA step, stop and alert instead
- Partial data saved before the expiry: mark the cutoff page so the next run resumes from there instead of restarting
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_rHfgJCipgea7fOLmNbhXeQ