scraper agent saved 200 profiles under the wrong candidates - pagination drifted when a result page re-ranked...
Fixes a scraper that saved profiles under the wrong candidates because the results page re-ranked mid-crawl and positional indexing drifted. Use when saved records do not match the profiles they claim to be. Key trigger: records whose stored data belongs to a different person than the stored identifier.
TL;DR: Key every saved profile by a stable identifier from the page (profile URL or ID), never by its position on the page. When a results page re-ranks mid-crawl, position 7 on page 2 is a different person than it was a minute ago. After keying by ID and re-verifying each mapping before saving, profiles land under the right candidates.
scraper agent saved 200 profiles under the wrong candidates - pagination drifted when a result page re-ranked mid-crawl- Reproduce the drift: save the result order of one page, wait, and re-fetch it. Expected: the order changed between fetches.
- Change the scraper to capture the profile ID or URL for each result along with its data. Expected: every record carries a stable key.
- Before saving, re-check that the ID on the stored record matches the ID on the page it was read from. Expected: mismatches get flagged instead of saved.
- Re-run the crawl on a small query and audit a sample of saved records. Expected: every sampled record's saved data matches the profile at its stored ID.
Use this when
- Saved candidate records contain another person's data
- The source site re-ranks or personalizes result order between requests
- A crawl ran long enough for the result set to shift underneath it
Not for this skill when
- The wrong data comes from parsing the wrong page section, not from reordering
- The site order is stable and records are wrong for another reason
- Profiles were never matched to candidates in the first place
Variant phrasings
Scraper mismatched profiles after page re-ranked mid-crawl
Pagination drift saved candidate data under wrong names
Crawl saved 200 profiles against the wrong candidate IDs
Why it happens
Indexing results by page-and-position assumes the listing is frozen during the crawl. Live sites re-rank constantly (freshness boosts, engagement signals, A/B tests), so the person at position N changes between the fetch and the save. The scraper then writes person B's data into the slot it reserved for person A.
Edge cases
- Sites with no visible profile ID in the listing: use the profile URL as the key, it is stable even when order is not
- Results that disappear between fetch and save: treat a missing ID as a skip, not as a reason to shift every later record up one slot
- The 200 already-saved wrong records: re-crawl with ID keying and overwrite, do not try to repair the drifted data in place
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_eULaaPUcy8ny0WSHGm-X7w
Maintainer review
No maintainer verification is recorded for this version.
This records the version a maintainer checked. It does not assert that the version is the latest upstream release.