agent's report scraper failed on salesforce lightning lazy-load error
Fixes a report scraper failing on Salesforce Lightning lazy loading: wait for network idle and data markers, not fixed sleeps. Use when scraped reports come back empty or partial. Not for API-based extracts.
TL;DR
Lightning lazy-loads report data as you scroll, so a scraper that reads immediately gets an empty shell. Wait for the network to go idle and for the row container to populate before extracting, scrolling to trigger loads. Better yet, use the analytics API and skip the scraper.
Error
Scraper failed: report table contained 0 rows; Lightning lazy-load never triggered.Steps
- Confirm lazy loading: the rows appear only after scrolling in a manual session. Expected: you verify the mechanism.
- In the scraper, wait for network idle after page load, then scroll the table container and wait again. Expected: rows populate.
- Wait for a data marker (row count text, a specific cell) rather than a fixed sleep. Expected: deterministic readiness.
- Extract after the marker appears. Expected: complete rows.
- Migrate to the analytics API for this report. Expected: the scraper becomes unnecessary.
When to use
- Scrapers return empty or partial Lightning reports.
- An agent scrapes Salesforce reports via browser.
- After Lightning updates change load behavior.
When not to use
- API extracts (no lazy loading involved).
- Reports that genuinely have no data (check filters).
Tool compatibility
- Playwright or Selenium with network-idle waits.
- Salesforce analytics API as the replacement.
Variant phrasings
lazy-load never triggered
Scroll and wait for network idle.
scraper got empty table
Same fix; wait for the data marker.
Why it happens
Lightning renders the chrome first and fetches data asynchronously, often gated on scroll position. Scrapers written against the final DOM miss the async phase.
Edge cases
- Virtualized tables render only visible rows; scroll through the whole table to capture all.
- Session timeouts during long scrapes produce partial data; keep scrapes short.
- The analytics API has row limits; page for large reports.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_BJvOzxXCwJzckK0EnQkwvA