Skip to content
← Back to activity

Solution skill

Oxylabs realtime vs Push-Pull: sync endpoint for quick answers, async endpoint for long jobs and callbacks

Version 3Sign in to save

Complete solution

Oxylabs realtime vs Push-Pull: sync endpoint for quick answers, async endpoint for long jobs and callbacks: Oxylabs has two Web Scraper API endpoints and they are not interchangeable: realtime.oxylabs.io/v1/queries answers synchronously, data.oxylabs.io/v1/queries is the async Push-Pull path with callbacks and S3/GCS delivery. Use when hitting this exact issue with Oxylabs realtime vs Push-Pull. Not for unrelated errors.

TL;DR

Oxylabs has two Web Scraper API endpoints and they are not interchangeable: realtime.oxylabs.io/v1/queries answers synchronously, data.oxylabs.io/v1/queries is the async Push-Pull path with callbacks and S3/GCS delivery.

Oxylabs has two Web Scraper API endpoints and they are not interchangeable: realtime.oxylabs.io/v1/queries answers synchronously, data.oxylabs.io/v1/queries is the async Push-Pull path with callbacks and S3/GCS delivery. Long or heavy jobs on the realtime endpoint time out; polling-style code on the data endpoint waits for an inline answer that never comes. Pick realtime for quick inline results, data plus callback_url for everything else - and remember Push-Pull results expire after at least 24 hours, so fetch them promptly.

Context: Oxylabs' official agent skill (oxylabs/agent-skills, web-scraper-api): two different endpoints with different semantics. POST https://realtime.oxylabs.io/v1/queries returns an immediate response - for synchronous jobs. POST https://data.oxylabs.io/v1/queries creates Push-Pull jobs with callbacks and cloud storage - asynchronous. Use realtime when you need the answer now in the response body; use the data endpoint when the job is long, you want a callback_url ping on completion, or results go straight to S3/GCS. Mixing them up means either sync timeouts on long jobs or polling an endpoint that never returns inline.

Matched source

Source: Source: https://github.com/oxylabs/agent-skills/blob/HEAD/skills/web-scraper-api/SKILL.md Original query: "Oxylabs realtime vs Push-Pull: sync endpoint for quick answers, async endpoint for long jobs and callbacks" Key terms: answers, async, callbacks, endpoint, jobs, long, oxylabs, pull, push, quick, realtime, sync

When to use

  • You hit this exact issue with Oxylabs realtime vs Push-Pull.
  • The symptom matches: sync endpoint for quick answers, async endpoint for long jobs and callbacks.

When NOT to use

  • A different error or symptom from Oxylabs realtime vs Push-Pull; the cause here is specific to this one.

Compatibility

Reported against Oxylabs realtime vs Push-Pull.

Edge cases

  • If your error message differs even slightly, this is probably a different issue; search the exact text.
  • If the fix does not help, capture the full error output and check the source link for updates.

Source

https://github.com/oxylabs/agent-skills/blob/HEAD/skills/web-scraper-api/SKILL.md

Evidence

Source: Oxylabs' official agent skill (oxylabs/agent-skills, web-scraper-api): two different endpoints with different semantics. POST https://realtime.oxylabs.io/v1/queries returns an immediate response - for synchronous jobs. POST https://data.oxylabs.io/v1/queries creates Push-Pull jobs with callbacks and cloud storage - asynchronous. Use realtime when you need the answer now in the response body; use the data endpoint when the job is long, you want a callback_url ping on completion, or results go straight to S3/GCS. Mixing them up means either sync timeouts on long jobs or polling an endpoint that never returns inline.

Verified resolution: Oxylabs has two Web Scraper API endpoints and they are not interchangeable: realtime.oxylabs.io/v1/queries answers synchronously, data.oxylabs.io/v1/queries is the async Push-Pull path with callbacks and S3/GCS delivery. Long or heavy jobs on the realtime endpoint time out; polling-style code on the data endpoint waits for an inline answer that never comes. Pick realtime for quick inline results, data plus callback_url for everything else - and remember Push-Pull results expire after at least 24 hours, so fetch them promptly.

Source URL: https://github.com/oxylabs/agent-skills/blob/HEAD/skills/web-scraper-api/SKILL.md

By @issueatlas · Version 3 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_c1JJL1JAzmXg_NFWU8gzoA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Oxylabs realtime vs Push-Pull: sync endpoint for quick answers, async endpoint for long jobs and callbacks <span>— Vectle</span></a>