docs agent hallucinated changelog entry, release notes broken
Fixes release notes containing changelog entries the agent invented that match no real commit. The LLM writer summarized imagined changes instead of reading the git history, so the notes describe work that never happened. Use when a changelog entry cannot be traced to a commit. Trigger: release notes mention a change with no matching commit or PR.
TL;DR
Derive every changelog entry from the actual git history and require each entry to cite its commit. The agent's writer invented plausible-sounding changes instead of reading commits, so the notes described work that never shipped. Fix: generate the entry list from git log between tags, attach the commit hash to each line, and reject any entry without a source.
The error
docs agent hallucinated changelog entry, release notes brokenSteps
Take the suspect entry and search for it in the git history:
git log --oneline v[PREV]..v[CURR] --grep="[keyword]"and check linked PRs. Expected: no commit or PR matches the entry, confirming it was invented.Delete the hallucinated entry and rebuild the notes from the real range:
git log --oneline v[PREV]..v[CURR]. Expected: every remaining entry maps to at least one commit in the range.Change the agent's writer prompt so it receives the commit list as its only source material and must cite the short hash for each entry it writes. Expected: the writer cannot produce an entry without pointing at a commit.
Add a verification pass: parse the generated notes, extract cited hashes, and check each one exists in the tag range with
git cat-file -t [hash]. Expected: any uncited or unknown hash fails the build.Re-run the release-notes job and review the diff. Expected: notes contain only real changes, each traceable to a commit.
Use this when
- a changelog entry describes a change with no matching commit or PR
- release notes were written by an agent from a summary rather than from git log
- reviewers flag notes that do not match what actually shipped
Not for this skill when
- entries are real but duplicated or mis-ordered (a formatting issue)
- the notes are missing entries rather than inventing them
- a human wrote the notes (an editing problem, not a pipeline problem)
Variant phrasings
- release notes describe changes that never happened
- agent invented changelog entries
- changelog entry has no corresponding commit
- hallucinated release notes from docs agent
Why it happens
An LLM writer asked to "write release notes for v2.4" without being grounded in the commit list will happily produce fluent, plausible entries from patterns in its training data and from the repo's general direction. Fluency is not evidence, and without a citation requirement there is nothing stopping invention.
Edge cases
- Squash-merged PRs: cite the PR number as well as the squash commit so the entry survives history rewrites.
- Entries carried over from a draft: verify them against the final tag range, not the draft's range.
- Breaking changes: require the writer to quote the breaking-change marker from the commit message, not paraphrase it.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_y3L2GKO-TrNLE3meJ-uu7g