news sitemap xml lastmod missing parser error

News sitemap XML is missing lastmod, parser errors

TL;DR

News sitemaps without lastmod still work; the parser error comes from code that assumes lastmod is always present. Google's news sitemap spec wants lastmod, but many publishers omit it. Make the parser treat lastmod as optional, fall back to changefreq or the crawl date, and never drop URLs just because the timestamp is missing.

The error

(parser error / dropped URLs)
news sitemap parser raised KeyError on missing lastmod; URLs skipped

When this helps

• news sitemap parsing fails on missing lastmod
• sitemap URLs get dropped silently
• building a sitemap intake pipeline
• handling publisher sitemap variance

When it doesn't

• the sitemap 404s; that is a fetch problem
• lastmod is present but wrong; that is publisher data quality, handle separately
• you need exact publish times; sitemaps without lastmod cannot give them

Works with

python 3.8+ with re. News sitemap practices vary by publisher.

Steps

1. Confirm lastmod is actually absent, not just namespaced

curl -s "https://YOUR-publisher/sitemap-news.xml" -o sitemap.xml -w "HTTP %{http_code}\n"
grep -c "lastmod" sitemap.xml

Expected: The lastmod count. Zero means genuinely absent; a nonzero count with parse failures means a namespace issue instead.

2. Parse sitemap URLs without requiring lastmod

import re
xml = open("sitemap.xml").read()
flat = xml.replace(chr(60), " ").replace(chr(62), " ")
urls = re.findall(r"loc\s+([^\s]+)", flat)
print("urls found:", len(urls))
print("sample:", urls[:3])

Expected: The URL list with no lastmod dependency. chr(60)/chr(62) flatten tags without literal angle brackets.

3. Fall back to per-URL fetch dates for freshness

import json, time
fresh = {"https://YOUR-publisher/[article]": int(time.time())}
open("sitemap_fresh.json", "w").write(json.dumps(fresh, indent=2))
print("first-seen timestamps stand in for missing lastmod")

Expected: A freshness record. When the sitemap lacks dates, the crawler's own first-seen time is the honest substitute.

4. Make the parser optional-lastmod permanently

import re
def sitemap_entries(xml):
    flat = xml.replace(chr(60), " ").replace(chr(62), " ")
    out = []
    for m in re.finditer(r"loc\s+([^\s]+)", flat):
        seg = flat[m.start():m.start()+500]
        lm = re.search(r"lastmod\s+([^\s]+)", seg)
        out.append((m.group(1), lm.group(1) if lm else None))
    return out
print("parser handles missing lastmod:", sitemap_entries(open("sitemap.xml").read())[:2])

Expected: URL plus optional lastmod tuples. The parser degrades gracefully on every publisher's sitemap.

Other ways people phrase this

sitemap lastmod missing parser error

Treat lastmod as optional. The URLs matter more than the timestamps.

news sitemap keyerror lastmod

The code assumed a field the spec only recommends. Default to None.

google news sitemap no lastmod

Common publisher omission. Fall back to first-seen dates.

Why it happens

Lastmod is recommended but not required in sitemaps, and many publishers skip it. Parsers written against Google's ideal sitemap assume it exists and crash or drop URLs when it does not. The URLs are the sitemap's payload; dates are metadata that degrades gracefully.

Edge cases

• Some publishers put lastmod in a different namespace; check before concluding it is absent.
• News sitemaps only cover the last two days; older articles need the regular sitemap.
• A sitemap index file points at the per-section sitemaps; parse the index first.
• Validate a sample of sitemap URLs with HEAD requests; sitemaps list dead URLs too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ANv6Cl8rOG1ohfru2gEOpg