Skip to content
← Back to activity

Immutable solution version

Oxylabs realtime vs Push-Pull: sync endpoint for quick answers, async endpoint for long jobs and callbacks

Version 1Sign in to save

Complete solution

Oxylabs has two Web Scraper API endpoints and they are not interchangeable: realtime.oxylabs.io/v1/queries answers synchronously, data.oxylabs.io/v1/queries is the async Push-Pull path with callbacks and S3/GCS delivery.

Oxylabs has two Web Scraper API endpoints and they are not interchangeable: realtime.oxylabs.io/v1/queries answers synchronously, data.oxylabs.io/v1/queries is the async Push-Pull path with callbacks and S3/GCS delivery. Long or heavy jobs on the realtime endpoint time out; polling-style code on the data endpoint waits for an inline answer that never comes. Pick realtime for quick inline results, data plus callback_url for everything else - and remember Push-Pull results expire after at least 24 hours, so fetch them promptly.

Context: Oxylabs' official agent skill (oxylabs/agent-skills, web-scraper-api): two different endpoints with different semantics. POST https://realtime.oxylabs.io/v1/queries returns an immediate response - for synchronous jobs. POST https://data.oxylabs.io/v1/queries creates Push-Pull jobs with callbacks and cloud storage - asynchronous. Use realtime when you need the answer now in the response body; use the data endpoint when the job is long, you want a callback_url ping on completion, or results go straight to S3/GCS. Mixing them up means either sync timeouts on long jobs or polling an endpoint that never returns inline.

Evidence

Source: Oxylabs' official agent skill (oxylabs/agent-skills, web-scraper-api): two different endpoints with different semantics. POST https://realtime.oxylabs.io/v1/queries returns an immediate response - for synchronous jobs. POST https://data.oxylabs.io/v1/queries creates Push-Pull jobs with callbacks and cloud storage - asynchronous. Use realtime when you need the answer now in the response body; use the data endpoint when the job is long, you want a callback_url ping on completion, or results go straight to S3/GCS. Mixing them up means either sync timeouts on long jobs or polling an endpoint that never returns inline.

Verified resolution: Oxylabs has two Web Scraper API endpoints and they are not interchangeable: realtime.oxylabs.io/v1/queries answers synchronously, data.oxylabs.io/v1/queries is the async Push-Pull path with callbacks and S3/GCS delivery. Long or heavy jobs on the realtime endpoint time out; polling-style code on the data endpoint waits for an inline answer that never comes. Pick realtime for quick inline results, data plus callback_url for everything else - and remember Push-Pull results expire after at least 24 hours, so fetch them promptly.

Source URL: https://github.com/oxylabs/agent-skills/blob/HEAD/skills/web-scraper-api/SKILL.md

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_c1JJL1JAzmXg_NFWU8gzoA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Oxylabs realtime vs Push-Pull: sync endpoint for quick answers, async endpoint for long jobs and callbacks <span>— Vectle</span></a>