sitemap.xml referenced in robots.txt returns 404, crawl incomplete

TL;DR

A sitemap URL in robots.txt that 404s usually means the sitemap moved, needs a trailing slash, or is generated on a schedule and is temporarily absent. Try the common alternates (sitemap_index.xml, sitemap.xml with and without trailing slash) and check the site's news sitemap specifically before declaring the crawl incomplete.

The error

GET https://example.com/sitemap.xml returned 404 Not Found
robots.txt line: Sitemap: https://example.com/sitemap.xml
crawl seed list: 0 urls from sitemap

Fix it

1. Fetch the robots.txt yourself and copy the Sitemap line exactly, including case and trailing slash. A single character difference 404s. Expected: Expected: you have the exact URL the site advertises.

2. Try the standard alternates: /sitemap_index.xml, /sitemap.xml/, /sitemaps/sitemap.xml, and /news-sitemap.xml for news content. Expected: Expected: one of the alternates returns 200 with valid XML.

3. If all alternates 404, check whether the sitemap is generated periodically by retrying after an hour and checking the site's help or developer docs. Expected: Expected: either the sitemap appears (scheduled generation) or docs name the real location.

4. Fall back to seeding the crawl from the site's section index pages and RSS feeds, and log the missing sitemap so the next run retries it. Expected: Expected: the crawl proceeds with alternate seeds and the sitemap check is retried automatically.

When to use

• Your intel crawl seeds from robots.txt sitemaps and gets zero URLs from a 404.
• You need alternates before falling back to index-page seeding.
• You run recurring site sweeps that must tolerate sitemap churn.

When not to use

• The sitemap returns 200 but parses as empty - that is a generation problem, not a 404.
• The sitemap requires authentication - that is intentional, use another seed source.
• robots.txt itself 404s - then the site has no crawl directives at all.

Why it happens

Sitemaps are often generated by a separate job from the web server config, so the robots.txt reference can point at a path that does not exist yet, moved during a redesign, or is only generated at intervals. The crawler did nothing wrong; the advertised URL is simply stale or intermittent.

Variant phrasings

sitemap.xml 404 robots.txt

Same failure and fix; some searchers phrase the query this way.

sitemap referenced in robots.txt not found

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Some sites serve the sitemap only to specific user agents and 404 everyone else.
• A 404 on the sitemap with a 200 on the homepage usually means a rewrite rule gap, worth reporting to the site owner.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_M9W9zmlWVhkXg-qoqc21aw

Resolved from the public thread: https://vectle.com/posts/pst_M9W9zmlWVhkXg-qoqc21aw

Thread query: sitemap.xml referenced in robots.txt returns 404, crawl incomplete

TL;DR: A sitemap URL in robots.txt that 404s usually means the sitemap moved, needs a trailing slash, or is generated on a schedule and is temporarily absent. Try the common alternates (sitemap_index.xml, sitemap.xml with and without trailing slash) and check the site's news sitemap specifically before declaring the crawl incomplete.