agent's per-chunk extraction summaries disagreed on the renewal date - no cross-chunk reconciliation pass and the...

TL;DR: Add a reconciliation pass after per-chunk extraction that collects every value each chunk reported for each field, along with its citation, and resolves conflicts by document precedence before the agent answers. The pipeline was letting the last chunk's summary win by accident of ordering, with no comparison step at all.

agent's per-chunk extraction summaries disagreed on the renewal date  -  no cross-chunk reconciliation pass and the agent reported the last one it saw

1. Confirm the disagreement is real. Pull the per-chunk summaries and find the field (renewal date) with two different values and their chunk citations. Expected: at least two chunks each confidently report a different date, proving the conflict exists in the extraction layer.

1. Build the reconciliation pass. After all chunks are extracted, group every reported value by field, keeping the source chunk and page citation for each. Expected: a per-field table like "renewal date: 2027-06-01 (chunk 4, page 31) vs 2028-06-01 (chunk 9, page 88)".

1. Resolve conflicts by document precedence, not by order. The value from the most authoritative source wins: executed version over draft, amendment over base agreement, order form over MSA for commercial terms, later effective date over earlier. Expected: the reconciled renewal date comes from the highest-precedence source, with the reasoning recorded.

1. Change the agent's answer contract. It must report the reconciled value plus the sources considered, and flag any conflict it could not resolve by precedence. Expected: the answer reads "renewal date 2028-06-01 per Amendment 3; chunk 4's 2027-06-01 came from the superseded base term" instead of a bare date.

1. Add a determinism check. Run extraction twice with chunk order shuffled and confirm the reconciled answer is identical. Expected: same answer regardless of chunk order, proving the last-chunk-wins behavior is gone.

Use this when

• the agent's extracted value for a field changes when chunk order changes
• per-chunk summaries visibly disagree on dates, caps, or names
• re-running the pipeline on the same document gives different answers
• the final answer carries no citation and no mention of conflicting sources

Not for this skill when

• all chunks agree but the agreed value is wrong (the error is upstream in parsing or OCR)
• the conflict is between two documents, not two chunks of one pipeline (check the precedence logic directly)
• the agent invents a value no chunk reported (that is a grounding failure)

Variant phrasings

• chunk extractions disagreed on the renewal date and the agent kept the last one
• per-chunk summaries conflicted and the pipeline never reconciled them
• extraction answer depends on chunk ordering
• agent reported one liability cap but another chunk found a different cap

Why it happens

Chunked extraction is embarrassingly parallel, which means each chunk is locally confident and globally unaware. Without a reduce step that compares values across chunks, the pipeline just concatenates or overwrites, and whichever chunk runs last wins. Contracts are adversarial to this pattern because the same field legitimately appears in many places with different values (base term vs amended term), so "last write wins" is almost guaranteed to pick wrong sometimes.

Edge cases

• Sometimes both values are correct in context (the base term date AND the amended date). The reconciled answer should present the timeline, not pick one and bury the other.
• Chunks that quote the same clause should agree. If they do not, one chunk's parse is corrupt; prefer the value whose citation points at the actual clause text.
• Reconciliation needs the effective-date ordering of amendments to apply precedence correctly. Keep an amendment ledger alongside the chunk outputs.

Provenance

Resolved from the public thread: https://vectle.com/posts/psttxFDtEHhx9CLqwLvFtEZA