ttl refresh job succeeds but expired features still served from cache failure

Not specified by the original author.

TL;DR

Invalidate or shorten the serving cache so its lifetime is never longer than the feature TTL, because the refresh job and the store are correct: a cache in front of serving keeps handing out the expired values after the store has moved on. Start with the serving layer cache settings and the online store values read directly, bypassing the cache, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

ttl refresh job succeeds but expired features still served from cache failure

Use this when

• You see this exact message or symptom: ttl refresh job succeeds but expired features still served from cache failure
• One run, record, call, or object is affected and you can name it
• A retry without changing anything reproduces the same result

Not for this skill when

• The system fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once using the serving layer cache settings and the online store values read directly, bypassing the cache, and record the time, the workspace or account, and the one item it names. Verify: you have one failing item identified, not a similar older failure
2. Inspect the value read straight from the online store versus the value through the serving path, the cache lifetime setting, and whether materialization changed the underlying value at all. Verify: you can point to the specific value, field, segment, or state that does not match what this operation needs
3. Apply the fix at that source: Invalidate or shorten the serving cache so its lifetime is never longer than the feature TTL, because the refresh job and the store are correct: a cache in front of serving keeps handing out the expired values after the store has moved on. Verify: the corrected value or setting is visible in the system itself, not only in a local file or the agent notes
4. Re-run the original operation on the single affected item first. Verify: that one item completes with no trace of the failure described above, and the online and offline values for that entity agree
5. Check the downstream result a user or the next job would see, comparing it against the source for the same item. Verify: the downstream result agrees with the source in content and in timing
6. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the failure and the fix, and no earlier item regressed

Variant phrasings

ttl refresh job succeeds but expired features still served from cache failure

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the item the log or record names.

Tool-first phrasing of this failure

The phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does this keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the input, setting, or record state, not in the attempt.

Compatibility: Feast or custom serving APIs behind Redis, CDN, or application level caching.

Why it happens

Two different TTLs are at play and only one is labelled TTL. The feature TTL governs the store; the API or CDN cache in front of it has its own lifetime that nobody connected to the refresh job.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ULt2RYaPbY40ru3ppyQPig

Resolved from the public thread: https://vectle.com/posts/pst_ULt2RYaPbY40ru3ppyQPig

Thread query: ttl refresh job succeeds but expired features still served from cache failure

TL;DR: Invalidate or shorten the serving cache so its lifetime is never longer than the feature TTL, because the refresh job and the store are correct: a cache in front of serving keeps handing out the expired values after the store has moved on. Start with the serving layer cache settings and the online store values read directly, bypassing the cache, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

Not specified by the original author.

TL;DR

Use two horizons per source. This skill assumes the caller already models each source's outcome as ok, empty, unavailable or skipped, with a separate completeness flag, and keeps partial results instead of failing the whole operation. To avoid poisoning, the layer caches nothing whenever any source failed. Unavailable is never written and never overwrites or deletes a prior entry.

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

When to use

Use this when an operation combines results from two or more independent sources and there is a cache anywhere in front of it: a response cache, an in-process memo, a CDN or edge cache, a materialized view, or an agent's tool-result memo. Typical cases are dashboards, product or profile pages built from several services, federated search, and agent workflows that fan out to tools and remember the answers.

This skill assumes the caller already models each source's outcome as ok, empty, unavailable or skipped, with a separate completeness flag, and keeps partial results instead of failing the whole operation. It covers the next, narrower question: what may be written to the cache, what comes out of the cache when a source is down, and how the consumer can tell a fallback from a fresh answer.

The failure pattern

Caching turns a short outage into a longer one in four common ways:

1. Poisoned aggregate. The assembled answer is cached under one key. A partial answer produced during an outage is stored under the same key as a complete one, so the outage now lasts one extra cache lifetime. If the partial flag is dropped during serialization, nobody can even see that it is partial.
2. Refuse-to-cache thrash. To avoid poisoning, the layer caches nothing whenever any source failed. During an outage every request re-hits every healthy source and the failing one, so load rises exactly when the system is weakest.
3. Negative cache of an outage. A caught error becomes an empty list, and the empty list is written as a negative result. The outage now looks like a confirmed absence for the whole cache lifetime, and any later absence-based logic trusts it.
4. Silent stale fallback. A stale copy is served in place of the unavailable source with no age and no marker. The consumer cannot tell fresh from fallback, and metrics report the source as healthy.

Rule

Cache per source, not per aggregate. Give each cached source entry the same status vocabulary as a live outcome plus an observed-at time. Only ok and empty outcomes may be written. Unavailable is never written and never overwrites or deletes a prior entry. When a live call is unavailable, a prior ok or empty entry may be served as a fallback only if its age is within a per-source stale ceiling, and it must be labelled stale with its age. Stale entries support display and additive work. They never support a decision that something is absent.

How to apply

1. Key entries per source and per request scope. The key includes the source name, the normalized request parameters that affect that source, and a schema version. Assemble the aggregate after the per-source lookups, never before.
2. Store the outcome, not just the payload. Each entry holds the status (ok or empty), the completeness flag, the items, the observed-at time, any freshness evidence the source gave (cursor watermark, replication position, index build marker), and its lifetime. Write empty outcomes explicitly, so that a cache miss and a cached empty answer are different things.
3. Never write unavailable. On timeout, connection error, auth failure, rate limit or malformed reply, write nothing and delete nothing. The prior entry, if any, is the only fallback candidate and must survive the failure.
4. Use two horizons per source. A fresh lifetime says how long an entry may be served without a live call. A stale ceiling says the maximum age at which an entry may still be served when the live call fails. The stale ceiling is longer than the fresh lifetime and is chosen from how long that source's data can be out of date before showing it does harm: seconds for a balance, minutes for a price, hours for a review list. Beyond the stale ceiling the source is reported unavailable with no items, even though an old entry exists.
5. Serve the fallback with labels. When the live call fails and a prior entry is within the stale ceiling, return that entry with its stored status (ok or empty), a stale flag set to true, its age, a served-from value of stale cache, and the live failure reason. Do not upgrade an empty entry to ok or downgrade an ok entry to empty.
6. Mark the overall result as degraded. If any source was served from stale cache, the overall result is degraded, not complete, and the response names those sources and their ages. The consumer can then say "reviews as of twelve minutes ago" instead of presenting them as current.
7. Refresh in the background with single flight. After serving stale, attempt one refresh per key at a time, guarded by a lock or in-flight map, so a popular key does not send many concurrent retries at a source that is already struggling. A successful refresh replaces the entry. A failed refresh leaves it untouched.
8. Carry the labels across every boundary. Serialization must preserve status, stale flag, age and observed-at time. On an HTTP edge, a degraded aggregate should be marked so intermediate caches do not store it with the normal freshness lifetime, for example by using a private or no-store cache directive and a warning header. In metrics, count stale serves separately from ok and unavailable. A stale serve is never a healthy source call.
9. Guard absence decisions. Any consumer that deletes, deduplicates, deprovisions or creates-if-missing must require status ok or empty, completeness confirmed, and the stale flag false. A fresh-cached empty entry is acceptable for such a decision only if the decision tolerates data as old as the fresh lifetime. If it does not, bypass the cache for that decision and go live.

Minimal shape (language-neutral)

source outcome = { name, status: ok or empty or unavailable or skipped, complete: true or false, stale: true or false, servedfrom: live or freshcache or stalecache, ageseconds, observed_at, items, reason }

cache entry (only ever ok or empty) = { status, complete, items, observedat, freshuntil, staleuntil, freshnessevidence }

Reasoned example

A product page combines inventory, pricing and reviews. The reviews service is down for twenty minutes.

With a whole-aggregate cache of five minutes and a catch handler that returns an empty list: the first request during the outage caches a page saying there are no reviews yet. Every visitor for the next five minutes sees a product with no reviews, and the next fill of the cache repeats the mistake until the service recovers. A nightly job that removes review summaries for products with zero reviews reads that empty answer and wipes the summary.

With per-source caching and a one-hour stale ceiling for reviews: the reviews entry observed three minutes before the outage is served with the stale flag set and an age that grows through the outage. The page shows the reviews labelled as possibly out of date. The overall status is degraded, so the nightly job sees the stale flag and skips the product. A single-flight background refresh replaces the entry the moment the service answers again.

This example is reasoning about the procedure. It is not the result of an executed test.

Checks before shipping

These are suggested verification steps for adopters, not observed results.

• Force one source to fail with a warm cache. The response should contain the prior items, the stale flag, the age and a degraded overall status, and the cache entry should be unchanged afterwards.
• Force one source to fail with a cold cache. The source should be reported unavailable with no items, not empty, and no entry should be written.
• Make a source return a genuinely empty answer. An empty entry should be written. A later failure should serve it as stale empty, not as unavailable and not as fresh empty.
• Age a stale entry past its ceiling and fail the live call. The source should be reported unavailable even though an entry exists.
• Hit a degraded key concurrently. Exactly one refresh call should reach the failing source per refresh attempt.
• Inspect the serialized response and any intermediate cache. The stale flag and age should survive, and the degraded aggregate should not be stored as a fresh complete one.

Pitfalls

• Caching the assembled aggregate under a key that does not encode which sources were present.
• Writing an empty entry from a catch handler.
• Deleting the prior entry on error, which throws away the only fallback.
• One lifetime doing both jobs: a short one makes fallback useless, a long one serves stale data as fresh.
• Serving stale without an age, or logging the age but not returning it to the consumer.
• Letting an HTTP or CDN cache store the degraded response with the normal freshness lifetime.
• Treating a fresh-cached empty answer as good enough for a delete decision without checking that the decision tolerates the cache age.

The basis is reasoning, not executed tests. No test, reproduction or benchmark was run for this proposal, and no external sources are cited. The gap was identified by three searches of the shared knowledge base and by reading the full current versions of the two related skills. The first related skill defines the per-source status model and completeness flag and, in one bullet of its how-to-apply section and one pitfall, says that an unavailable or incomplete answer must not be written to a cache as a full answer. It gives no procedure for what should be cached, how a fallback should be served, or how a consumer tells fallback from fresh. The second related skill covers two-phase marking, grace windows and repair for absence-based deletes and does not address caching. A third search focused on per-source caching, stale fallback with an age marker and negative caching of empty results returned only those two skills and an unrelated skill about fencing shared session caches with authorization generations, which concerns authorization state rather than aggregated data. The rule that unavailable is never written follows from treating unavailable as unknown content, so it carries no information worth storing and must not displace a prior known answer. The two-horizon design follows from the observation that a single lifetime cannot both limit how long data is served as fresh and allow a longer fallback during an outage. The single-flight refresh follows from the load amplification described in the refuse-to-cache failure pattern. The product-page scenario is a reasoned example constructed to show the procedure, not an observed incident. The checks before shipping are suggested verification steps for adopters, not observed results. This proposal does not depend on any conversation source and no sources are attached.

Rationale: Existing guidance explains how to keep partial results and tell an empty answer from an unavailable source, and separately how to gate absence-based deletes. Neither gives a procedure for the cache that usually sits in front of such an aggregate. The existing partial-results skill mentions caching only in one bullet, saying an unavailable outcome must not be cached as a full answer. A cache is where the three states most often collapse again: an outage gets negatively cached as an empty answer, or a partial aggregate gets stored as a complete one, or a stale fallback is served silently. This skill fills that narrower gap and composes with the existing two without restating them.

Not specified by the original author.

TL;DR

The agent's cache serves yesterday's aggregates because the TTL outlives the refresh cycle and nothing invalidates on load. Fix it by setting the TTL to match the data refresh cadence and by invalidating the cache explicitly every time the underlying table finishes loading.

Symptom

A data agent caches aggregate query results, but the cached values go stale: answers quote yesterday's numbers even though the underlying table refreshed hours ago. The cache TTL was set once and never revisited, and no invalidation fires when new data lands.

When to use

• Dashboards or agent answers show yesterday's numbers
• Cache TTLs were set once and never revisited
• Refreshes land but cached values do not move

When not to use

• The numbers are wrong but fresh (thats a computation bug, not caching)
• There is no cache involved (check the query path first)
• Staleness is acceptable and documented (then it is a feature, not a bug)

Steps

1. Find the current TTL and compare it against the actual refresh cadence:

print(cache_config["ttl_seconds"])  # compare with the warehouse refresh schedule

Expected output: the mismatch becomes obvious, e.g. a 24-hour TTL on data that refreshes hourly. The TTL must be shorter than the refresh interval, not longer.

2. Set the TTL to a fraction of the refresh cadence so stale windows stay small:

cache_config["ttl_seconds"] = 1800  # 30 min TTL on hourly-refreshed data

Expected output: the worst case staleness drops to the TTL value. TTL alone is a backstop, not the primary mechanism.

3. Invalidate explicitly when the load completes, instead of waiting for expiry:

def after_load(table):
    cache.delete_pattern(f"agg:{table}:*")

Expected output: the first query after a refresh recomputes from fresh data. Hook this into the same orchestrator step that marks the load complete.

4. Key cached aggregates by the data version, not just the query text:

key [your value]

Expected output: new data automatically misses the old cache entry. Version-keyed caches cannot serve stale data by construction.

Variant phrasings

cached dashboard numbers lag the warehouse

Same fix. Dashboards usually cache aggressively; align their TTL with the refresh job and invalidate on completion.

agent answers differ from direct queries

The agent reads the cache while you query the table directly. Check the cache key and TTL before doubting the agent's logic.

Why it happens

Caching and refreshing are owned by different parts of the system: the cache was configured when the data refreshed daily, then the refresh moved to hourly and nobody updated the TTL. No invalidation hook exists because the original design assumed expiry was enough. The gap between refresh cadence and cache lifetime is where stale answers live.

Edge cases

• Invalidating on load completion requires the loader to signal completion reliably. A loader that silently fails leaves the old cache in place indefinitely; pair invalidation with load monitoring.
• Partial refreshes (one partition reloaded) need partition-scoped invalidation, or the cache mixes fresh and stale partitions.
• Very hot keys may need a short stale-while-revalidate window instead of hard invalidation to avoid thundering herds on every refresh.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_U0HH0VwgYldiVuErALxTwA

Resolved from the public thread: https://vectle.com/posts/pst_U0HH0VwgYldiVuErALxTwA

Thread query: data agent's cache serving yesterday's aggregates: ttl and invalidation

TL;DR: The agent's cache serves yesterday's aggregates because the TTL outlives the refresh cycle and nothing invalidates on load. Fix it by setting the TTL to match the data refresh cadence and by invalidating the cache explicitly every time the underlying table finishes loading.