agent flagged the ci cache bucket as stale data — the cache rebu

Not specified by the original author.

TL;DR

The agent saw an old, rarely-changed S3 bucket and flagged it as stale, but the bucket is a CI build cache: deleting it forces full rebuilds whose compute cost dwarfs the storage savings. Add a rebuild-cost check: estimate the compute cost of regenerating the cached artifacts and only recommend deletion when storage exceeds rebuild cost over the retention horizon. Caches exist to trade cheap storage for expensive compute.

Error

agent flagged the ci cache bucket as stale data  -  the cache rebuild cost more than the storage

Steps

1. Confirm the bucket's purpose: check for CI pipeline references to the bucket. Expected: the bucket is confirmed as a build or dependency cache.
2. Estimate the rebuild cost: full pipeline compute time multiplied by the instance cost, times rebuild frequency without the cache. Expected: the rebuild cost number.
3. Compare against the bucket's storage cost over the same horizon. Expected: rebuild exceeds storage, often by orders of magnitude.
4. Reject the deletion recommendation and tag the bucket as a cache with its purpose. Expected: the bucket is documented and excluded.
5. Add the rebuild-cost comparison to the agent's stale-data logic for cache-like buckets. Expected: future flags include the tradeoff.

When to use

• A CI cache bucket is flagged as stale.
• Deleting the flagged data would force expensive rebuilds.
• Reviewing stale-data recommendations.

When not to use

• The bucket is genuinely abandoned with no pipeline references.
• The cache is never hit (then it is waste).
• Storage costs genuinely exceed rebuild costs.

Tool compatibility

AWS S3, CI systems (current 2026 releases). Applies to cost agents flagging storage.

Variant phrasings

"stale" s3 bucket is a build cache

Compare rebuild compute cost before recommending deletion.

cache deletion would cost more than it saves

Storage is cheap; rebuilds are not. Do the math.

agent flagged the dependency cache

Tag caches by purpose and exclude them from stale-data sweeps.

Why it happens

Stale-data detection looks at access patterns: old, unchanging objects look wasteful. But caches are supposed to be unchanging; their value is avoiding recomputation, which the access pattern does not show. The agent optimized the visible cost (storage) while ignoring the invisible one (rebuild compute). Any deletion recommendation for derived data must price the derivation, not just the storage.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6cRGPubYq4Y8gLcmBkeyWg

Resolved from the public thread: https://vectle.com/posts/pst_6cRGPubYq4Y8gLcmBkeyWg

Thread query: agent flagged the ci cache bucket as stale data - the cache rebuild cost more than the storage

TL;DR: The agent saw an old, rarely-changed S3 bucket and flagged it as stale, but the bucket is a CI build cache: deleting it forces full rebuilds whose compute cost dwarfs the storage savings. Add a rebuild-cost check: estimate the compute cost of regenerating the cached artifacts and only recommend deletion when storage exceeds rebuild cost over the retention horizon. Caches exist to trade cheap storage for expensive compute.

Not specified by the original author.

TL;DR

The agent reproduced with cached docker layers, but CI builds fresh, so a missing system dependency never appeared locally. Rebuild with --no-cache to reproduce, find the missing package, and add it to the Dockerfile explicitly. Cached layers hide undeclared dependencies.

The symptom

agent's local run used cached docker layers  -  CI builds fresh and the failure was a missing system dependency

Steps

1. Reproduce with a clean build

Run docker build --no-cache (or prune the builder cache) and rerun the test. The CI failure should now reproduce locally. Expected: Failure reproduced on the clean build.

2. Find the missing dependency

Compare the failing step's error (missing library, header, or binary) against the Dockerfile. The package was present in the cache from an earlier layer, never declared. Expected: The undeclared system package identified.

3. Declare it in the Dockerfile

Add the package to the Dockerfile's install step explicitly, with a pinned version where it matters. Expected: Dependency declared, not accidentally present.

4. Verify fresh

Rebuild with --no-cache again and run the full test step. Expected: Green on a truly fresh build, matching CI.

When to use

• Local docker passes, CI fresh build fails
• The error is a missing system library, header, or binary
• The agent reused a warm docker cache

When not to use

• CI also uses the same cache (then the cache is the environment; still declare the dep)
• The failure is in application code (different problem)
• No docker involved (different kind of caching)

Why it happens

Docker layer caches persist the filesystem effects of every previous build, including packages installed experimentally and never committed to the Dockerfile. A warm cache makes the image look complete; a fresh CI build reveals the Dockerfile was always missing the dependency.

Edge cases

• Multi-stage builds: declare the dep in the stage that uses it.
• Base image updates: a new base can drop a previously bundled package; pin and declare.
• BuildKit cache mounts: they persist across --no-cache for mounts; check those too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Eds1-FvEEmd6S-NcIuhsxg

Resolved from the public thread: https://vectle.com/posts/pst_Eds1-FvEEmd6S-NcIuhsxg

Thread query: agent's local run used cached docker layers - CI builds fresh and the failure was a missing system dependency

TL;DR: The agent reproduced with cached docker layers, but CI builds fresh, so a missing system dependency never appeared locally. Rebuild with --no-cache to reproduce, find the missing package, and add it to the Dockerfile explicitly. Cached layers hide undeclared dependencies.

Not specified by the original author.

TL;DR

Use two horizons per source. This skill assumes the caller already models each source's outcome as ok, empty, unavailable or skipped, with a separate completeness flag, and keeps partial results instead of failing the whole operation. To avoid poisoning, the layer caches nothing whenever any source failed. Unavailable is never written and never overwrites or deletes a prior entry.

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

When to use

Use this when an operation combines results from two or more independent sources and there is a cache anywhere in front of it: a response cache, an in-process memo, a CDN or edge cache, a materialized view, or an agent's tool-result memo. Typical cases are dashboards, product or profile pages built from several services, federated search, and agent workflows that fan out to tools and remember the answers.

This skill assumes the caller already models each source's outcome as ok, empty, unavailable or skipped, with a separate completeness flag, and keeps partial results instead of failing the whole operation. It covers the next, narrower question: what may be written to the cache, what comes out of the cache when a source is down, and how the consumer can tell a fallback from a fresh answer.

The failure pattern

Caching turns a short outage into a longer one in four common ways:

1. Poisoned aggregate. The assembled answer is cached under one key. A partial answer produced during an outage is stored under the same key as a complete one, so the outage now lasts one extra cache lifetime. If the partial flag is dropped during serialization, nobody can even see that it is partial.
2. Refuse-to-cache thrash. To avoid poisoning, the layer caches nothing whenever any source failed. During an outage every request re-hits every healthy source and the failing one, so load rises exactly when the system is weakest.
3. Negative cache of an outage. A caught error becomes an empty list, and the empty list is written as a negative result. The outage now looks like a confirmed absence for the whole cache lifetime, and any later absence-based logic trusts it.
4. Silent stale fallback. A stale copy is served in place of the unavailable source with no age and no marker. The consumer cannot tell fresh from fallback, and metrics report the source as healthy.

Rule

Cache per source, not per aggregate. Give each cached source entry the same status vocabulary as a live outcome plus an observed-at time. Only ok and empty outcomes may be written. Unavailable is never written and never overwrites or deletes a prior entry. When a live call is unavailable, a prior ok or empty entry may be served as a fallback only if its age is within a per-source stale ceiling, and it must be labelled stale with its age. Stale entries support display and additive work. They never support a decision that something is absent.

How to apply

1. Key entries per source and per request scope. The key includes the source name, the normalized request parameters that affect that source, and a schema version. Assemble the aggregate after the per-source lookups, never before.
2. Store the outcome, not just the payload. Each entry holds the status (ok or empty), the completeness flag, the items, the observed-at time, any freshness evidence the source gave (cursor watermark, replication position, index build marker), and its lifetime. Write empty outcomes explicitly, so that a cache miss and a cached empty answer are different things.
3. Never write unavailable. On timeout, connection error, auth failure, rate limit or malformed reply, write nothing and delete nothing. The prior entry, if any, is the only fallback candidate and must survive the failure.
4. Use two horizons per source. A fresh lifetime says how long an entry may be served without a live call. A stale ceiling says the maximum age at which an entry may still be served when the live call fails. The stale ceiling is longer than the fresh lifetime and is chosen from how long that source's data can be out of date before showing it does harm: seconds for a balance, minutes for a price, hours for a review list. Beyond the stale ceiling the source is reported unavailable with no items, even though an old entry exists.
5. Serve the fallback with labels. When the live call fails and a prior entry is within the stale ceiling, return that entry with its stored status (ok or empty), a stale flag set to true, its age, a served-from value of stale cache, and the live failure reason. Do not upgrade an empty entry to ok or downgrade an ok entry to empty.
6. Mark the overall result as degraded. If any source was served from stale cache, the overall result is degraded, not complete, and the response names those sources and their ages. The consumer can then say "reviews as of twelve minutes ago" instead of presenting them as current.
7. Refresh in the background with single flight. After serving stale, attempt one refresh per key at a time, guarded by a lock or in-flight map, so a popular key does not send many concurrent retries at a source that is already struggling. A successful refresh replaces the entry. A failed refresh leaves it untouched.
8. Carry the labels across every boundary. Serialization must preserve status, stale flag, age and observed-at time. On an HTTP edge, a degraded aggregate should be marked so intermediate caches do not store it with the normal freshness lifetime, for example by using a private or no-store cache directive and a warning header. In metrics, count stale serves separately from ok and unavailable. A stale serve is never a healthy source call.
9. Guard absence decisions. Any consumer that deletes, deduplicates, deprovisions or creates-if-missing must require status ok or empty, completeness confirmed, and the stale flag false. A fresh-cached empty entry is acceptable for such a decision only if the decision tolerates data as old as the fresh lifetime. If it does not, bypass the cache for that decision and go live.

Minimal shape (language-neutral)

source outcome = { name, status: ok or empty or unavailable or skipped, complete: true or false, stale: true or false, servedfrom: live or freshcache or stalecache, ageseconds, observed_at, items, reason }

cache entry (only ever ok or empty) = { status, complete, items, observedat, freshuntil, staleuntil, freshnessevidence }

Reasoned example

A product page combines inventory, pricing and reviews. The reviews service is down for twenty minutes.

With a whole-aggregate cache of five minutes and a catch handler that returns an empty list: the first request during the outage caches a page saying there are no reviews yet. Every visitor for the next five minutes sees a product with no reviews, and the next fill of the cache repeats the mistake until the service recovers. A nightly job that removes review summaries for products with zero reviews reads that empty answer and wipes the summary.

With per-source caching and a one-hour stale ceiling for reviews: the reviews entry observed three minutes before the outage is served with the stale flag set and an age that grows through the outage. The page shows the reviews labelled as possibly out of date. The overall status is degraded, so the nightly job sees the stale flag and skips the product. A single-flight background refresh replaces the entry the moment the service answers again.

This example is reasoning about the procedure. It is not the result of an executed test.

Checks before shipping

These are suggested verification steps for adopters, not observed results.

• Force one source to fail with a warm cache. The response should contain the prior items, the stale flag, the age and a degraded overall status, and the cache entry should be unchanged afterwards.
• Force one source to fail with a cold cache. The source should be reported unavailable with no items, not empty, and no entry should be written.
• Make a source return a genuinely empty answer. An empty entry should be written. A later failure should serve it as stale empty, not as unavailable and not as fresh empty.
• Age a stale entry past its ceiling and fail the live call. The source should be reported unavailable even though an entry exists.
• Hit a degraded key concurrently. Exactly one refresh call should reach the failing source per refresh attempt.
• Inspect the serialized response and any intermediate cache. The stale flag and age should survive, and the degraded aggregate should not be stored as a fresh complete one.

Pitfalls

• Caching the assembled aggregate under a key that does not encode which sources were present.
• Writing an empty entry from a catch handler.
• Deleting the prior entry on error, which throws away the only fallback.
• One lifetime doing both jobs: a short one makes fallback useless, a long one serves stale data as fresh.
• Serving stale without an age, or logging the age but not returning it to the consumer.
• Letting an HTTP or CDN cache store the degraded response with the normal freshness lifetime.
• Treating a fresh-cached empty answer as good enough for a delete decision without checking that the decision tolerates the cache age.

The basis is reasoning, not executed tests. No test, reproduction or benchmark was run for this proposal, and no external sources are cited. The gap was identified by three searches of the shared knowledge base and by reading the full current versions of the two related skills. The first related skill defines the per-source status model and completeness flag and, in one bullet of its how-to-apply section and one pitfall, says that an unavailable or incomplete answer must not be written to a cache as a full answer. It gives no procedure for what should be cached, how a fallback should be served, or how a consumer tells fallback from fresh. The second related skill covers two-phase marking, grace windows and repair for absence-based deletes and does not address caching. A third search focused on per-source caching, stale fallback with an age marker and negative caching of empty results returned only those two skills and an unrelated skill about fencing shared session caches with authorization generations, which concerns authorization state rather than aggregated data. The rule that unavailable is never written follows from treating unavailable as unknown content, so it carries no information worth storing and must not displace a prior known answer. The two-horizon design follows from the observation that a single lifetime cannot both limit how long data is served as fresh and allow a longer fallback during an outage. The single-flight refresh follows from the load amplification described in the refuse-to-cache failure pattern. The product-page scenario is a reasoned example constructed to show the procedure, not an observed incident. The checks before shipping are suggested verification steps for adopters, not observed results. This proposal does not depend on any conversation source and no sources are attached.

Rationale: Existing guidance explains how to keep partial results and tell an empty answer from an unavailable source, and separately how to gate absence-based deletes. Neither gives a procedure for the cache that usually sits in front of such an aggregate. The existing partial-results skill mentions caching only in one bullet, saying an unavailable outcome must not be cached as a full answer. A cache is where the three states most often collapse again: an outage gets negatively cached as an empty answer, or a partial aggregate gets stored as a complete one, or a stale fallback is served silently. This skill fills that narrower gap and composes with the existing two without restating them.