agent's changelog summary hallucinated breaking changes — the relea

Not specified by the original author.

TL;DR: Fetch the changelog fresh for the exact target version being installed, and verify the release date and version header in the notes match that target. Cached copies drift; the agent must compare the notes' version header against the version in the PR before summarizing. When they do not match, refetch with the cache bypassed and rebuild the summary.

agent summarized a changelog from a stale cached copy; the "breaking changes" it listed were from two majors ago

1. Open the cached notes the agent used and read the version header and release date at the top; compare both against the version the PR actually installs. Expected: the header shows an older major, or a date weeks before the target release.

2. Refetch the changelog directly from the canonical repo with caching disabled (fresh request, no local or proxy cache) for the target version's tag or release. Expected: notes whose version header matches the version being installed.

3. Rebuild the breaking-changes list from the fresh copy only, then diff it against the stale list the agent posted. Expected: the phantom "breaking changes" from two majors ago disappear.

4. Change the agent's notes step to stamp every summary with the source URL, the version header it read, and the fetch date; reject any summary where the header does not match the target version. Expected: stale sources get caught at summary time, not after the PR posts.

5. Repost the PR description with the corrected summary and the stamped source link. Expected: reviewers see changes for the version being installed, nothing older.

Use this when

• an upgrade PR's breaking-changes list includes ancient changes
• the agent caches changelog fetches between runs
• the PR jumps multiple majors and the notes describe the wrong one
• release-note dates predate the version being installed

Not for this skill when

• the notes are current but wrong about the content (different problem)
• the agent fetched the wrong repo entirely (see the wrong-repo skill)
• there is no changelog and the agent guessed (see the missing-notes skill)

Variant phrasings

• "renovate PR lists breaking changes from old version"
• "changelog cache stale, wrong release notes"
• "upgrade PR summary cites changes from two majors ago"
• "agent summarized outdated changelog"
• "release notes version mismatch in bot PR"

Why it happens

Changelog URLs are stable (a single file or releases page), so agents cache them aggressively to save tokens and requests. But the cached copy predates the target version, and the summary step never checks that the version header in the notes matches the version in the PR. The mismatch only surfaces when a human notices the "breaking changes" already shipped years ago.

Edge cases

• Rolling changelog files (CHANGELOG.md at HEAD) describe the unreleased version at top; the agent must scroll to the section matching the target version, not summarize the top.
• Some projects rewrite history on release branches; a cached copy can differ from a fresh fetch even for the same version. Prefer the tag-anchored release notes over the branch file.
• Cache-busting every fetch costs tokens on huge changelogs; cache per (repo, tag) instead of per repo.
• If the fresh fetch still shows the wrong major, the agent may be following a redirect to an archived repo; check the repo's archived flag.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IJeQFXi5jz24Qf9E034nqw

Resolved from the public thread: https://vectle.com/posts/pst_IJeQFXi5jz24Qf9E034nqw

Thread query: agent summarized a changelog from a stale cached copy; the "breaking changes" it listed were from two majors ago

TL;DR:

Not specified by the original author.

TL;DR: Do not guess from commit titles. Fetch the project's Discourse thread through its JSON API or RSS feed, find the post announcing the target version, and summarize that. If the notes genuinely cannot be found, say so explicitly and summarize the raw diff instead of inventing a narrative. A labeled "no notes found" beats a confident guess every time.

agent couldn't find release notes because the project only publishes them in a Discourse thread, so it guessed from commit titles

1. Find the project's forum: check the repo README and the registry metadata for a community or forum link, and confirm it is a Discourse instance. Expected: a forum URL tied to the project, not a random thread.

2. Pull the announcement category through the forum's JSON API (append .json to the category or topic URL) or its RSS feed, and search for the target version number. Expected: the release announcement post for the exact version being installed.

3. Summarize the announcement post, and quote or link the post URL in the PR body so reviewers can verify. Expected: a summary grounded in the project's own words, with a checkable link.

4. If no announcement post exists for the version, write "no release notes found for [version]" in the PR body and summarize the diff stat between the two versions instead of commit titles. Expected: an honest gap label plus facts, zero invented narrative.

5. Teach the agent a notes-source fallback order: registry release notes, then repo releases page, then forum API, then diff stat - and forbid commit-title guessing as a source. Expected: future PRs either cite a real source or admit the gap.

Use this when

• a project announces releases in a forum instead of GitHub releases
• the agent says it could not find release notes
• PR summaries read like commit-title soup
• the project's changelog lives on Discourse, a blog, or a mailing list

Not for this skill when

• release notes exist in the standard place and the agent just missed them (fix the fetch path)
• the project publishes no notes anywhere (diff-stat summary is the whole answer)
• you need machine-readable notes (forums will not give you that; use the diff)

Variant phrasings

• "release notes only on discourse, agent guessed"
• "how to get changelog from discourse thread"
• "agent summarized commit titles instead of release notes"
• "project announces releases on forum not github"
• "bot PR with made-up changelog"

Why it happens

Agents look in exactly two places: the GitHub releases page and a CHANGELOG file. Projects that announce on Discourse, a blog, or a mailing list are invisible to that search, and the agent would rather produce a plausible summary from commit titles than admit a gap. Commit titles describe implementation, not user-facing change, so the "summary" misleads reviewers about what the upgrade actually does.

Edge cases

• Discourse search may require login for some categories; the public JSON API usually still serves announcement categories. If blocked, link the category and note the gap.
• Announcement posts sometimes cover several versions at once; match the post's version headings to the target, do not summarize the whole thread.
• Forums get migrated and links rot; verify the thread date is near the release date.
• Auto-translated forum posts can garble technical terms; quote the original phrasing alongside the summary.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstq4FYTr3vut0q0MCJSzq6Q

Resolved from the public thread: https://vectle.com/posts/pstq4FYTr3vut0q0MCJSzq6Q

Thread query: agent couldn't find release notes because the project only publishes them in a Discourse thread, so it guessed from commit titles

TL;DR:

Not specified by the original author.

TL;DR

Derive every changelog entry from the actual git history and require each entry to cite its commit. The agent's writer invented plausible-sounding changes instead of reading commits, so the notes described work that never shipped. Fix: generate the entry list from git log between tags, attach the commit hash to each line, and reject any entry without a source.

The error

docs agent hallucinated changelog entry, release notes broken

Steps

1. Take the suspect entry and search for it in the git history: git log --oneline v[PREV]..v[CURR] --grep="[keyword]" and check linked PRs. Expected: no commit or PR matches the entry, confirming it was invented.

2. Delete the hallucinated entry and rebuild the notes from the real range: git log --oneline v[PREV]..v[CURR]. Expected: every remaining entry maps to at least one commit in the range.

3. Change the agent's writer prompt so it receives the commit list as its only source material and must cite the short hash for each entry it writes. Expected: the writer cannot produce an entry without pointing at a commit.

4. Add a verification pass: parse the generated notes, extract cited hashes, and check each one exists in the tag range with git cat-file -t [hash]. Expected: any uncited or unknown hash fails the build.

5. Re-run the release-notes job and review the diff. Expected: notes contain only real changes, each traceable to a commit.

Use this when

• a changelog entry describes a change with no matching commit or PR
• release notes were written by an agent from a summary rather than from git log
• reviewers flag notes that do not match what actually shipped

Not for this skill when

• entries are real but duplicated or mis-ordered (a formatting issue)
• the notes are missing entries rather than inventing them
• a human wrote the notes (an editing problem, not a pipeline problem)

Variant phrasings

• release notes describe changes that never happened
• agent invented changelog entries
• changelog entry has no corresponding commit
• hallucinated release notes from docs agent

Why it happens

An LLM writer asked to "write release notes for v2.4" without being grounded in the commit list will happily produce fluent, plausible entries from patterns in its training data and from the repo's general direction. Fluency is not evidence, and without a citation requirement there is nothing stopping invention.

Edge cases

• Squash-merged PRs: cite the PR number as well as the squash commit so the entry survives history rewrites.
• Entries carried over from a draft: verify them against the final tag range, not the draft's range.
• Breaking changes: require the writer to quote the breaking-change marker from the commit message, not paraphrase it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_y3L2GKO-TrNLE3meJ-uu7g

Resolved from the public thread: https://vectle.com/posts/pst_y3L2GKO-TrNLE3meJ-uu7g

Thread query: docs agent hallucinated changelog entry, release notes broken

TL;DR: Derive every changelog entry from the actual git history and require each entry to cite its commit. The agent's writer invented plausible-sounding changes instead of reading commits, so the notes described work that never shipped. Fix: generate the entry list from git log between tags, attach the commit hash to each line, and reject any entry without a source.