scraping the PR files page returned empty: agent's selector missed the virtualized list

TL;DR: GitHub's files list is virtualized - rows only exist in the DOM near the viewport, so a selector that runs too early (or against the wrong container) finds nothing. Wait for the first real row selector with an explicit timeout, scroll in increments to hydrate more rows, and collect rows as they appear. And keep the paginated files API as the fallback, because the DOM will always be flakier than the API.

scraping the PR files page returned empty: agent's selector missed the virtualized list

1. Confirm the virtualization. Open the files page, inspect the list container, and scroll: watch rows mount and unmount as the viewport moves.

Expected: you see rows appearing and disappearing with scroll, proving the full list is never in the DOM at once.

1. Fix the wait. Replace any fixed sleep with an explicit wait for a concrete row selector (the per-file row element), with a timeout around 15 to 30 seconds.

Expected: the wait succeeds when rows hydrate and fails fast with a clear timeout when they do not.

1. Check the selector itself against the live DOM. Virtualized lists often nest rows inside a scroll container with generated class names; verify your selector matches actual row elements in the current page markup.

Expected: the selector matches at least the visible rows in a manual check.

1. Scroll to hydrate. Scroll the list container in viewport-sized increments, pausing briefly after each, and collect newly appeared rows each time until a full pass adds no new rows.

Expected: the collected set grows with each scroll pass and then stabilizes.

1. Dedupe the collected rows by file path, since virtualization can re-mount the same row.

Expected: the final list has one entry per file path.

1. Add the API fallback. If the row wait times out or the collected count looks wrong, fetch the paginated files API instead and log that the scrape was abandoned.

Expected: the run still produces a complete file list even when the DOM path fails.

Use this when

• Scraping the PR files page returns empty but files are visible
• The selector worked last month and finds nothing now (markup changed)
• Only the first few rows are ever collected
• You need DOM rows but the list virtualizes

Not for this skill when

• The page never finishes loading at all (that is a page-load hang, different fix)
• GitHub serves a rate-limit interstitial instead of the page (rate-limit problem)
• You only need the file list, not rendered rows (use the files API directly and skip the browser)
• The browser automation times out mid-scroll rather than finding nothing (scroll-timeout problem)

Variant phrasings

• "GitHub files list virtualized, scraper finds no rows"
• "selector returns empty on PR files changed tab"
• "scrape virtualized diff list GitHub"
• "files page rows not in DOM for automation"

Why it happens

Virtualized lists render only the rows near the viewport to keep huge PRs fast. A scraper that queries the DOM once - especially before the list hydrates - sees an empty container. It is not that the selector is wrong in principle; it ran at a moment (or against a container state) where no rows existed yet.

Edge cases

• GitHub changes its markup without notice. Pin a periodic selector health check rather than assuming the selector stays valid.
• Rows for very large files may render a placeholder instead of a full row. Decide whether placeholders count as collected or need a second pass.
• Headless rendering can hydrate slower than headed. What works on your dev machine may need longer waits in CI.
• If the PR has thousands of files, full scroll hydration is slow and flaky. Prefer the API for the list and reserve the DOM for the handful of rows you actually need to interact with.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dURnwM7rX1TF7EIAMzOOIg