# News sitemap XML is missing lastmod, parser errors

## TL;DR
News sitemaps without lastmod still work; the parser error comes from code that assumes lastmod is always present. Google's news sitemap spec wants lastmod, but many publishers omit it. Make the parser treat lastmod as optional, fall back to changefreq or the crawl date, and never drop URLs just because the timestamp is missing.

## The error
```text
(parser error / dropped URLs)
news sitemap parser raised KeyError on missing lastmod; URLs skipped
```

## When this helps
- news sitemap parsing fails on missing lastmod
- sitemap URLs get dropped silently
- building a sitemap intake pipeline
- handling publisher sitemap variance

## When it doesn't
- the sitemap 404s; that is a fetch problem
- lastmod is present but wrong; that is publisher data quality, handle separately
- you need exact publish times; sitemaps without lastmod cannot give them

## Works with
python 3.8+ with re. News sitemap practices vary by publisher.

## Steps
### 1. Confirm lastmod is actually absent, not just namespaced
```bash
curl -s "https://YOUR-publisher/sitemap-news.xml" -o sitemap.xml -w "HTTP %{http_code}\n"
grep -c "lastmod" sitemap.xml
```
Expected: The lastmod count. Zero means genuinely absent; a nonzero count with parse failures means a namespace issue instead.

### 2. Parse sitemap URLs without requiring lastmod
```python
import re
xml = open("sitemap.xml").read()
flat = xml.replace(chr(60), " ").replace(chr(62), " ")
urls = re.findall(r"loc\s+([^\s]+)", flat)
print("urls found:", len(urls))
print("sample:", urls[:3])
```
Expected: The URL list with no lastmod dependency. chr(60)/chr(62) flatten tags without literal angle brackets.

### 3. Fall back to per-URL fetch dates for freshness
```python
import json, time
fresh = {"https://YOUR-publisher/[article]": int(time.time())}
open("sitemap_fresh.json", "w").write(json.dumps(fresh, indent=2))
print("first-seen timestamps stand in for missing lastmod")
```
Expected: A freshness record. When the sitemap lacks dates, the crawler's own first-seen time is the honest substitute.

### 4. Make the parser optional-lastmod permanently
```python
import re
def sitemap_entries(xml):
    flat = xml.replace(chr(60), " ").replace(chr(62), " ")
    out = []
    for m in re.finditer(r"loc\s+([^\s]+)", flat):
        seg = flat[m.start():m.start()+500]
        lm = re.search(r"lastmod\s+([^\s]+)", seg)
        out.append((m.group(1), lm.group(1) if lm else None))
    return out
print("parser handles missing lastmod:", sitemap_entries(open("sitemap.xml").read())[:2])
```
Expected: URL plus optional lastmod tuples. The parser degrades gracefully on every publisher's sitemap.

## Other ways people phrase this
### sitemap lastmod missing parser error
Treat lastmod as optional. The URLs matter more than the timestamps.

### news sitemap keyerror lastmod
The code assumed a field the spec only recommends. Default to None.

### google news sitemap no lastmod
Common publisher omission. Fall back to first-seen dates.

## Why it happens
Lastmod is recommended but not required in sitemaps, and many publishers skip it. Parsers written against Google's ideal sitemap assume it exists and crash or drop URLs when it does not. The URLs are the sitemap's payload; dates are metadata that degrades gracefully.

## Edge cases
- Some publishers put lastmod in a different namespace; check before concluding it is absent.
- News sitemaps only cover the last two days; older articles need the regular sitemap.
- A sitemap index file points at the per-section sitemaps; parse the index first.
- Validate a sample of sitemap URLs with HEAD requests; sitemaps list dead URLs too.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ANv6Cl8rOG1ohfru2gEOpg