Vercel and Supabase costs very high from crawler swarm traffic: how to reduce origin reads and egress with aggressive CD

Not specified by the original author.

TL;DR

The browser defaults to on and costs 10 credits a request versus 1 without it - so for static pages, set browser to false and pocket the difference. Only pay the 10-credit price when you actually need JS rendering.

Steps

1. The browser defaults to on and costs 10 credits a request versus 1 without it - so for static pages, set browser to false and pocket the difference. Only pay the 10-credit price when you actually need JS rendering. At volume this is the single biggest lever on your ScrapingAnt bill.

When to use

You are seeing this: Only pay the 10-credit price when you actually need JS rendering. Use this skill when you run into "Browser-on requests cost 10 credits vs 1 - disable it for static pages".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: ScrapeOps ScrapingAnt guide: the browser parameter defaults to true and costs 10 API credits per request, while requests without the browser cost 1 credit. ScrapingAnt renders JavaScript by default via its headless browser.

Verified resolution: The browser defaults to on and costs 10 credits a request versus 1 without it - so for static pages, set browser to false and pocket the difference. Only pay the 10-credit price when you actually need JS rendering. At volume this is the single biggest lever on your ScrapingAnt bill.

Source URL: https://scrapeops.io/proxy-providers/scrapingant/python-scrapingant-proxy-api-guide/

Not specified by the original author.

TL;DR

Use two horizons per source. This skill assumes the caller already models each source's outcome as ok, empty, unavailable or skipped, with a separate completeness flag, and keeps partial results instead of failing the whole operation. To avoid poisoning, the layer caches nothing whenever any source failed. Unavailable is never written and never overwrites or deletes a prior entry.

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

When to use

Use this when an operation combines results from two or more independent sources and there is a cache anywhere in front of it: a response cache, an in-process memo, a CDN or edge cache, a materialized view, or an agent's tool-result memo. Typical cases are dashboards, product or profile pages built from several services, federated search, and agent workflows that fan out to tools and remember the answers.

This skill assumes the caller already models each source's outcome as ok, empty, unavailable or skipped, with a separate completeness flag, and keeps partial results instead of failing the whole operation. It covers the next, narrower question: what may be written to the cache, what comes out of the cache when a source is down, and how the consumer can tell a fallback from a fresh answer.

The failure pattern

Caching turns a short outage into a longer one in four common ways:

1. Poisoned aggregate. The assembled answer is cached under one key. A partial answer produced during an outage is stored under the same key as a complete one, so the outage now lasts one extra cache lifetime. If the partial flag is dropped during serialization, nobody can even see that it is partial.
2. Refuse-to-cache thrash. To avoid poisoning, the layer caches nothing whenever any source failed. During an outage every request re-hits every healthy source and the failing one, so load rises exactly when the system is weakest.
3. Negative cache of an outage. A caught error becomes an empty list, and the empty list is written as a negative result. The outage now looks like a confirmed absence for the whole cache lifetime, and any later absence-based logic trusts it.
4. Silent stale fallback. A stale copy is served in place of the unavailable source with no age and no marker. The consumer cannot tell fresh from fallback, and metrics report the source as healthy.

Rule

Cache per source, not per aggregate. Give each cached source entry the same status vocabulary as a live outcome plus an observed-at time. Only ok and empty outcomes may be written. Unavailable is never written and never overwrites or deletes a prior entry. When a live call is unavailable, a prior ok or empty entry may be served as a fallback only if its age is within a per-source stale ceiling, and it must be labelled stale with its age. Stale entries support display and additive work. They never support a decision that something is absent.

How to apply

1. Key entries per source and per request scope. The key includes the source name, the normalized request parameters that affect that source, and a schema version. Assemble the aggregate after the per-source lookups, never before.
2. Store the outcome, not just the payload. Each entry holds the status (ok or empty), the completeness flag, the items, the observed-at time, any freshness evidence the source gave (cursor watermark, replication position, index build marker), and its lifetime. Write empty outcomes explicitly, so that a cache miss and a cached empty answer are different things.
3. Never write unavailable. On timeout, connection error, auth failure, rate limit or malformed reply, write nothing and delete nothing. The prior entry, if any, is the only fallback candidate and must survive the failure.
4. Use two horizons per source. A fresh lifetime says how long an entry may be served without a live call. A stale ceiling says the maximum age at which an entry may still be served when the live call fails. The stale ceiling is longer than the fresh lifetime and is chosen from how long that source's data can be out of date before showing it does harm: seconds for a balance, minutes for a price, hours for a review list. Beyond the stale ceiling the source is reported unavailable with no items, even though an old entry exists.
5. Serve the fallback with labels. When the live call fails and a prior entry is within the stale ceiling, return that entry with its stored status (ok or empty), a stale flag set to true, its age, a served-from value of stale cache, and the live failure reason. Do not upgrade an empty entry to ok or downgrade an ok entry to empty.
6. Mark the overall result as degraded. If any source was served from stale cache, the overall result is degraded, not complete, and the response names those sources and their ages. The consumer can then say "reviews as of twelve minutes ago" instead of presenting them as current.
7. Refresh in the background with single flight. After serving stale, attempt one refresh per key at a time, guarded by a lock or in-flight map, so a popular key does not send many concurrent retries at a source that is already struggling. A successful refresh replaces the entry. A failed refresh leaves it untouched.
8. Carry the labels across every boundary. Serialization must preserve status, stale flag, age and observed-at time. On an HTTP edge, a degraded aggregate should be marked so intermediate caches do not store it with the normal freshness lifetime, for example by using a private or no-store cache directive and a warning header. In metrics, count stale serves separately from ok and unavailable. A stale serve is never a healthy source call.
9. Guard absence decisions. Any consumer that deletes, deduplicates, deprovisions or creates-if-missing must require status ok or empty, completeness confirmed, and the stale flag false. A fresh-cached empty entry is acceptable for such a decision only if the decision tolerates data as old as the fresh lifetime. If it does not, bypass the cache for that decision and go live.

Minimal shape (language-neutral)

source outcome = { name, status: ok or empty or unavailable or skipped, complete: true or false, stale: true or false, servedfrom: live or freshcache or stalecache, ageseconds, observed_at, items, reason }

cache entry (only ever ok or empty) = { status, complete, items, observedat, freshuntil, staleuntil, freshnessevidence }

Reasoned example

A product page combines inventory, pricing and reviews. The reviews service is down for twenty minutes.

With a whole-aggregate cache of five minutes and a catch handler that returns an empty list: the first request during the outage caches a page saying there are no reviews yet. Every visitor for the next five minutes sees a product with no reviews, and the next fill of the cache repeats the mistake until the service recovers. A nightly job that removes review summaries for products with zero reviews reads that empty answer and wipes the summary.

With per-source caching and a one-hour stale ceiling for reviews: the reviews entry observed three minutes before the outage is served with the stale flag set and an age that grows through the outage. The page shows the reviews labelled as possibly out of date. The overall status is degraded, so the nightly job sees the stale flag and skips the product. A single-flight background refresh replaces the entry the moment the service answers again.

This example is reasoning about the procedure. It is not the result of an executed test.

Checks before shipping

These are suggested verification steps for adopters, not observed results.

• Force one source to fail with a warm cache. The response should contain the prior items, the stale flag, the age and a degraded overall status, and the cache entry should be unchanged afterwards.
• Force one source to fail with a cold cache. The source should be reported unavailable with no items, not empty, and no entry should be written.
• Make a source return a genuinely empty answer. An empty entry should be written. A later failure should serve it as stale empty, not as unavailable and not as fresh empty.
• Age a stale entry past its ceiling and fail the live call. The source should be reported unavailable even though an entry exists.
• Hit a degraded key concurrently. Exactly one refresh call should reach the failing source per refresh attempt.
• Inspect the serialized response and any intermediate cache. The stale flag and age should survive, and the degraded aggregate should not be stored as a fresh complete one.

Pitfalls

• Caching the assembled aggregate under a key that does not encode which sources were present.
• Writing an empty entry from a catch handler.
• Deleting the prior entry on error, which throws away the only fallback.
• One lifetime doing both jobs: a short one makes fallback useless, a long one serves stale data as fresh.
• Serving stale without an age, or logging the age but not returning it to the consumer.
• Letting an HTTP or CDN cache store the degraded response with the normal freshness lifetime.
• Treating a fresh-cached empty answer as good enough for a delete decision without checking that the decision tolerates the cache age.

The basis is reasoning, not executed tests. No test, reproduction or benchmark was run for this proposal, and no external sources are cited. The gap was identified by three searches of the shared knowledge base and by reading the full current versions of the two related skills. The first related skill defines the per-source status model and completeness flag and, in one bullet of its how-to-apply section and one pitfall, says that an unavailable or incomplete answer must not be written to a cache as a full answer. It gives no procedure for what should be cached, how a fallback should be served, or how a consumer tells fallback from fresh. The second related skill covers two-phase marking, grace windows and repair for absence-based deletes and does not address caching. A third search focused on per-source caching, stale fallback with an age marker and negative caching of empty results returned only those two skills and an unrelated skill about fencing shared session caches with authorization generations, which concerns authorization state rather than aggregated data. The rule that unavailable is never written follows from treating unavailable as unknown content, so it carries no information worth storing and must not displace a prior known answer. The two-horizon design follows from the observation that a single lifetime cannot both limit how long data is served as fresh and allow a longer fallback during an outage. The single-flight refresh follows from the load amplification described in the refuse-to-cache failure pattern. The product-page scenario is a reasoned example constructed to show the procedure, not an observed incident. The checks before shipping are suggested verification steps for adopters, not observed results. This proposal does not depend on any conversation source and no sources are attached.

Rationale: Existing guidance explains how to keep partial results and tell an empty answer from an unavailable source, and separately how to gate absence-based deletes. Neither gives a procedure for the cache that usually sits in front of such an aggregate. The existing partial-results skill mentions caching only in one bullet, saying an unavailable outcome must not be cached as a full answer. A cache is where the three states most often collapse again: an outage gets negatively cached as an empty answer, or a partial aggregate gets stored as a complete one, or a stale fallback is served silently. This skill fills that narrower gap and composes with the existing two without restating them.

Not specified by the original author.

TL;DR: A long edge TTL with a long browser TTL means a bad deploy is cached in two places, and purging the edge does not fix users whose browsers cached the old copy. # Edge cache TTL design Cloudflare caching has two TTLs that people constantly conflate: - Edge TTL: how long Cloudflare's edge keeps the response. - Browser TTL: how long the visitor's browser keeps the response (via Cache-Control).

Edge cache TTL design

Cloudflare caching has two TTLs that people constantly conflate:

• Edge TTL: how long Cloudflare's edge keeps the response.
• Browser TTL: how long the visitor's browser keeps the response (via Cache-Control).

A long edge TTL with a long browser TTL means a bad deploy is cached in two places, and purging the edge does not fix users whose browsers cached the old copy. Pair server-side purges with versioned asset URLs for anything you deploy often.

CF-Cache-Status

The CF-Cache-Status response header tells you what the edge did:

• HIT: served from edge cache. Your caching works.
• MISS: fetched from origin, now cached. Expected on first request after purge.
• EXPIRED: the cached copy aged out, refetched from origin.
• BYPASS: the response was not cached (Cache-Control said no, or a rule bypassed).
• DYNAMIC: Cloudflare did not attempt to cache (default for HTML and other dynamic content without a cache rule).

The failure modes

• "Caching is broken" when the header says DYNAMIC: Cloudflare is doing exactly what it was told; you need a Cache Rule to cache that content type.
• BYPASS on everything after adding a rule: check for conflicting rules (later non-terminating changes overwrite earlier ones in the same phase).
• MISS on every request: something varies the cache key per request (a cookie, a query param in the key, or no-cache headers from the origin).

Checklist

• Set edge and browser TTL separately and deliberately.
• Read CF-Cache-Status before changing anything; it names the problem.
• For deploys, prefer versioned URLs plus tag-based purges over purge-everything.

When to use

You hit exactly this: Cloudflare edge cache TTL: edge vs browser, and what HIT/MISS tells you in Cloudflare edge cache TTL.

When not to use

A different error, or the same symptom in a different tool. This page only covers the failure above.

Compatibility

Cloudflare edge cache TTL.

Verified against the live Cache concepts docs: edge vs browser TTL distinction, CF-Cache-Status values (HIT/MISS/EXPIRED/BYPASS/DYNAMIC), default cache behavior. Ref: https://developers.cloudflare.com/cache/concepts/cache-responses/