agent's Behance scraper kept missing the newest projects - the portfolio grid re-sorts by 'featured' on every load...

TL;DR: Pin the sort order to a stable option like newest first, or dedupe by project ID across pages. The grid re-sorts by "featured" on every load, so page 2 of this load is not page 2 of the last load and items slip through. After pinning the sort, pagination covers the full list.

agent's Behance scraper kept missing the newest projects  -  the portfolio grid re-sorts by 'featured' on every load and pagination skipped items

1. Load the grid twice and compare the order of the first page. Expected: the order differs between loads, confirming the re-sort.
2. Find a stable sort option in the page filters or URL (newest or published date). Expected: switching to it keeps the same order across loads.
3. Update the scraper to use the stable sort and to record each project's ID as it pages. Expected: a second run over the same pages finds zero new IDs it missed.
4. Add a dedupe check: if a page returns IDs already seen, stop instead of continuing. Expected: the scraper terminates cleanly at the end of the list instead of looping.

Use this when

• Scraped results keep missing the newest items on a portfolio grid
• Page order changes between loads on a site with featured or algorithmic sorting
• Pagination seems to skip items or return duplicates across pages

Not for this skill when

• The grid order is stable and items are missing for another reason (filters, lazy loading, login walls)
• The site offers no stable sort option and no API (then key everything by ID and crawl until the ID set stops growing)
• The problem is missing images or media, not missing items

Variant phrasings

Behance scraper missing new projects due to grid re-sorting

Pagination skipping items on featured-sorted grid

Scraper loses newest portfolio items between page loads

Why it happens

"Featured" and other editorial sorts are recomputed per request, often with randomization or engagement weighting mixed in. Pagination assumes a fixed order, so when the order shifts between page fetches, some items move to already-visited pages and others jump ahead of the crawler. The crawl completes but the coverage has holes.

Edge cases

• New projects published mid-crawl: with a newest-first sort they appear on page 1 and get picked up on the next run, not the current one
• Sites with no stable sort at all: crawl by ID set convergence (keep paging until no new IDs appear) instead of page numbers
• Infinite-scroll grids: scroll and capture IDs until the scroll stops yielding new ones, then apply the same dedupe logic

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dPH2yiEM7wSoZ05eKiiJyQ