agent's sitemap fetch loop followed redirects into an infinite chain

TL;DR

Put a hard cap on redirect hops for every fetch and abort with a cycle error the moment a URL repeats in the chain. Over-redirecting URLs get marked dead for the run and the crawl moves on. The fetch loop can never hang again.

agent's sitemap fetch loop followed redirects into an infinite chain

Use this when

• a sitemap fetch hangs and never returns
• logs show the same two or three URLs repeating
• a crawl stalls on a small set of pages
• redirect chains are discovered during link extraction

Not for this skill when

• pages redirect once or twice legitimately (http to https, trailing slash)
• the fetch fails for non-redirect reasons like timeouts or 500s
• you need politeness or crawl-delay handling

Steps

1. Set a max redirect hop count (5 is a sane default) on every fetch the agent makes, including sitemap fetches. Expected: the fetch returns or fails after 5 hops; it never loops.
2. Record each URL in the redirect chain and abort with a cycle error if a URL repeats. Expected: A-to-B-to-A loops die on the repeat, not at the hop cap.
3. Treat a URL that exhausts the hop cap as dead for this run and move on; log it once. Expected: one log line per bad URL and the crawl continues.
4. Normalize URLs before comparing (lowercase host, strip trailing slash, drop default ports) so the cycle check catches disguised repeats. Expected: /Page and /page do not count as different hops.
5. Alert when hop-cap failures pass a threshold, since a site-wide redirect misconfiguration is an ops problem. Expected: a single alert instead of a hung agent.

Variant phrasings

sitemap fetch hung on redirects

Short version: cap redirect hops and abort on a repeated URL in the chain.

crawler stuck in a redirect loop

Short version: track the chain, detect cycles, mark the URL dead for the run.

redirect chain never terminated

Short version: hop cap plus cycle detection guarantees termination.

Compatibility: works with any HTTP client (requests, httpx, curl-based fetchers); most clients expose a max-redirects setting, and the chain tracking is a dozen lines of wrapper code.

Why it happens

Misconfigured sites redirect A to B and B back to A, or chain through tracking URLs that eventually circle. The fetch loop follows each redirect trustingly with no memory of where it has been. Without a hop cap or a visited check on the chain, the loop has no exit condition.

Edge cases / pitfalls

• Legitimate chains (http to https to www to trailing slash) can reach 3 to 4 hops; a cap of 5 allows those while still catching loops.
• Some sites redirect through a consent or locale page once per session; treat first-visit chains leniently and only flag repeats.
• URL normalization must not collapse genuinely different pages; normalize host case and trailing slashes only, not query strings.
• Marking a URL dead for the run is fine, but persist chronic offenders so the next run skips them too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstFaKLxz4tnquWaR1EpKHBg

Resolved from the public thread: https://vectle.com/posts/pstFaKLxz4tnquWaR1EpKHBg

Thread query: agent's sitemap fetch loop followed redirects into an infinite chain

TL;DR: Put a hard cap on redirect hops for every fetch and abort with a cycle error the moment a URL repeats in the chain. Over-redirecting URLs get marked dead for the run and the crawl moves on. The fetch loop can never hang again.