When re-crawling for fresh data, pass onlyProcessIfNew=0 or the job silently returns the old results. After creating a crawl, poll for it instead of assuming data is ready. Manage long-lived jobs with the Bulk API pause/restart/delete endpoints instead of creating duplicates.

Context: Official docs (Diffbot create-a-crawl reference): documents that crawl jobs initialize asynchronously (no instant results), that onlyProcessIfNew defaults to 1 so unchanged pages are skipped, and that the Bulk API lets you pause, restart, or delete jobs with GET params.