map-reduce summaries of a 300-page portfolio contradicted each other and the agent picked the wrong one without...

TL;DR: Require every map summary to carry page citations and a confidence note, then make the reduce step reconcile contradictions explicitly instead of voting silently. If the reduce step cannot resolve a conflict from the citations, it must surface the conflict to the user. The agent was picking whichever summary sounded most confident, with no page numbers to check.

map-reduce summaries of a 300-page portfolio contradicted each other and the agent picked the wrong one without citing pages

1. Reproduce the contradiction with citations. Re-run the map step on the conflicting sections and require each summary to include page numbers for every factual claim. Expected: you can now see which pages each summary drew on, and the disagreement becomes checkable.

1. Change the map output contract. Each map summary must include: the claims, the page or section cited for each claim, and a note on anything ambiguous or partially visible in its chunk. Expected: map outputs are no longer bare assertions; every claim has a pointer back to source text.

1. Rewrite the reduce step as an explicit reconciliation pass. When two map summaries disagree, the reducer must compare their citations, prefer the citation that is more specific (exact clause over vague reference), and record which source won and why. Expected: the final summary explains conflicts ("Section 8 says X per page 112; the conflicting summary cited only the exhibit index") instead of hiding them.

1. Add a no-silent-pick rule. If the reducer cannot resolve a contradiction from the citations, the output must say so and present both versions with their pages. Expected: unresolved conflicts appear in the output as flagged disagreements, never as a quietly chosen winner.

1. Test with a known-conflict portfolio. Keep a small portfolio where two documents genuinely disagree (e.g. different renewal dates in the MSA vs an order form) and run the pipeline against it. Expected: the output surfaces the disagreement with both citations instead of picking one.

Use this when

• portfolio summaries contradict themselves across runs or across sections
• the agent's summary answers cite no pages or sections
• the reduce step merges map outputs by majority vote or by confidence score alone
• different documents in the portfolio genuinely disagree and the agent reports only one version

Not for this skill when

• there is a single document and the summary contradicts the source (check extraction or chunking instead)
• the map summaries agree but are all wrong in the same way (the error is upstream, in parsing or OCR)
• the agent hallucinates claims with fake page numbers (that is a grounding failure, not a reconciliation failure)

Variant phrasings

• map-reduce contract summaries disagreed and the agent chose without evidence
• portfolio summary picked the wrong renewal date from conflicting section summaries
• reduce step silently resolved contradictory clause extractions
• agent summarized 300 pages but cited nothing and contradicted itself

Why it happens

Map-reduce treats each chunk as independent, but contract portfolios are not independent: the same term appears in multiple documents with different values, and each map summary is locally confident. A reduce step that merges by confidence or recency is really just picking the loudest voice. Citations are the only thing that lets the reducer (or the human) adjudicate, because the right answer is the one tied to the operative clause, not the one stated most firmly.

Edge cases

• Genuine conflicts are normal in portfolios (order forms override MSAs). The reducer should know the precedence hierarchy: executed version beats draft, order form beats base agreement on pricing, later effective date beats earlier.
• Page numbers shift between versions. Citations should include document name plus page, or better, section numbers, which survive re-pagination.
• A map chunk that straddles a section boundary can produce a confidently wrong summary. Clause-aware chunking upstream reduces these; the citation requirement catches the rest.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0kCx8yF0ym32fmdxzbyhkA