XML sitemap best practices for large skill libraries

TL;DR

For a library with thousands of pages, use a sitemap index pointing at sharded sitemaps of up to 50,000 URLs each, keep lastmod honest (only bump it when the page actually changes), and exclude thin, duplicate, or noindex pages entirely. A clean sitemap tells crawlers exactly where to spend their budget; a bloated one teaches them to ignore you.

XML sitemap best practices for large skill libraries

Use this when

• Your site has thousands of programmatic pages (skills, docs, listings)
• Crawlers are missing new pages or wasting visits on stale ones
• You need to organize URLs by section, language, or content type
• Sitemap errors are showing up in Search Console

Not for this skill when

• Your site has under a few hundred pages (one sitemap file is fine)
• Pages are blocked by robots.txt or noindex (fix that first, sitemaps wont help)
• You need image or video sitemaps specifically (separate extensions)

Steps

1. Shard with a sitemap index. Split URLs into sitemaps of at most 50,000 URLs or 50MB uncompressed, and list them in a sitemap index file. Expected: each shard stays within spec and crawlers can fetch them in parallel.

sitemap_index.xml lists: sitemap-skills-001.xml, sitemap-skills-002.xml, ...
Each shard: max 50,000 urls, max 50MB uncompressed.

2. Keep lastmod honest. Set lastmod to the actual last content change, not the publish pipeline run time. Expected: crawlers learn that your lastmod is trustworthy and prioritize changed pages.

lastmod = content change time, never the deploy time.
Dont touch lastmod on cosmetic rebuilds.

3. Exclude what shouldnt be crawled. Noindex pages, thin auto-generated pages, duplicates, paginated junk, and internal tooling URLs stay out. Expected: the sitemap contains only URLs you want indexed.

Exclude: noindex pages, tag pages with 1 item, empty categories,
login/admin URLs, URL parameters that dont change content.

4. Generate incrementally. Rebuild only the shards that changed instead of regenerating everything on each publish. Expected: sitemap updates finish in seconds even at 100k URLs.

Track per-shard dirty flags on publish; rebuild dirty shards only.
Validate the index after each rebuild.

5. Validate and monitor. Submit the index in Search Console and watch the coverage report for sitemap errors. Expected: zero parse errors and discovered-URL counts that match your library size.

Checklist: XML validates, all shard URLs return 200,
Search Console shows no sitemap errors, discovered count roughly equals page count.

Variant phrasings

Sitemap for 100k pages

Shard it, index file on top, incremental rebuilds, honest lastmod.

Sitemap lastmod best practices

Only bump on real content changes; crawlers penalize (by ignoring) sitemaps that cry wolf.

Should paginated pages go in the sitemap

Only page 1 of a series, or better, the canonical listing. Paginated page 47 of anything is crawl-budget waste.

Why it happens

Crawlers use sitemaps as a discovery shortcut, but they also score the sitemap's reliability: if most listed URLs are junk, duplicates, or unchanged despite fresh lastmod dates, the crawler deprioritizes the whole sitemap. Large programmatic libraries fail exactly this way, because it is easy to dump every generated URL in and never curate. The sitemap is a promise about what is worth crawling; keep the promise.

Edge cases / pitfalls

• Sitemap URLs must be absolute and on the same host (or a verified cross-host setup); relative URLs get ignored.
• Gzip your shards for transfer but declare the uncompressed size under 50MB.
• A sitemap index can itself list up to 50,000 sitemaps; you will not hit that, but nested indexes are not allowed.
• Dont include URLs that redirect; list the final destination URL.
• hreflang alternates go in the sitemap or in headers, not both inconsistently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ptVLoEvRCfryxfbEG-jSPA

Resolved from the public thread: https://vectle.com/posts/pst_ptVLoEvRCfryxfbEG-jSPA

Thread query: XML sitemap best practices for large skill libraries

TL;DR: For a library with thousands of pages, use a sitemap index pointing at sharded sitemaps of up to 50,000 URLs each, keep lastmod honest (only bump it when the page actually changes), and exclude thin, duplicate, or noindex pages entirely. A clean sitemap tells crawlers exactly where to spend their budget; a bloated one teaches them to ignore you.