## TL;DR
An agent that re-fetches on every ETag change loops forever on pages whose ETag changes per request (load balancers, timestamps in headers). Compare content hashes instead of ETags, and cap re-fetch retries so a volatile page cannot spin the agent.

## The error
```text
fetch loop: 47 requests to https://news.example.com/live-blog
ETag differed each time; content identical (sha256 match).
Agent burned the page budget on one URL.
```

## Fix it
1. Hash the response body (sha256) on each fetch and compare hashes, not ETags. Identical hashes mean no new content regardless of headers.
   Expected: Expected: the second fetch shows a matching hash and the agent stops.
2. Add a per-URL fetch cap (for example 3 attempts) with a clear 'volatile page' log message when the cap is hit.
   Expected: Expected: the agent moves on after 3 fetches instead of looping to 47.
3. For live blogs and tickers that genuinely update, switch to a poll interval (5-15 minutes) instead of change-triggered refetch.
   Expected: Expected: the agent revisits on a schedule with bounded total requests.
4. Record which domains show ETag volatility and default them to hash comparison permanently.
   Expected: Expected: the pipeline learns the volatile domains and stops wasting requests.

## When to use
- Your intel agent burns its fetch budget re-requesting one volatile page.
- You need change detection that survives lying ETags.
- You monitor live blogs and updating pages.

## When not to use
- The content actually changes each fetch - then refetching is correct, just rate-limit it.
- The server sends no ETag at all - then use Last-Modified or content hash from the start.
- You need every intermediate version - then store each hash, but still cap the rate.

## Why it happens
Some servers generate a new ETag per response (embedding timestamps or node IDs), so the ETag never matches even when the body is byte-identical. An agent that treats ETag-change as content-change refetches forever. The content hash is the only reliable change signal.

## Variant phrasings
### agent loop refetching etag changed
Same failure and fix; some searchers phrase the query this way.
### etag keeps changing fetch loop fix
Same failure and fix; some searchers phrase the query this way.

## Edge cases
- Compressed versus uncompressed bodies hash differently; normalize (decompress first) before comparing.
- A page can alternate between two identical variants (A/B test); hash comparison handles this, ETag does not.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O_IV_ndXgbPgHaZkrCfuPg
