summary agent dropped the customer's actual question: extraction check

TL;DR

Summarizers optimize for brevity, and the customer's question is often one sentence in a long thread, so it gets compressed out. The handoff then answers everything except what was asked. The fix is an extraction check: pull the explicit question out first, keep it verbatim in the summary, and verify it's there before the summary ships.

The query

summary agent dropped the customer's actual question: extraction check

Use this when

• Summaries miss the customer's actual question
• Agents act on summaries and miss the point
• You're designing summarization for support handoffs

Not for

• Translation quality issues
• Summary tone problems
• Summary length tuning

Steps

1. Extract the question before summarizing

First pass: find the customer's explicit question or request and copy it verbatim. Second pass: summarize the context around it. The question leads the summary; it never gets compressed.

Expected output: every summary opening with the customer's verbatim question.

2. Verify the question survived

After generating the summary, check that the extracted question appears in it. If not, the summary fails validation and gets regenerated. This is the extraction check proper: a test the summary must pass.

Expected output: a validation step rejecting summaries that drop the question.

3. Handle implicit questions

Not every ask ends with a question mark: 'my invoice is wrong' is a request to fix it. Train the extractor on implicit asks too, and when unsure, include the customer's own words rather than an interpretation.

Expected output: implicit asks captured in the customer's own words.

4. Show the question to the next agent prominently

Format the handoff so the question is impossible to miss: first line, bolded or labeled. A perfect extraction buried at the bottom of the summary still gets missed.

Expected output: handoffs with the question as the first, prominent line.

Variant phrasings

ai summary missed customer question

Steps 1 and 2: extract first, then verify it survived.

support handoff summary incomplete

Step 4's formatting; the question must lead.

Why it happens

Summarization is lossy compression, and the loss isn't random: models keep the frequent (context, backstory) and drop the rare (the one-sentence ask). The question is the highest-value sentence in the thread and the most likely to be compressed away. Extracting it first inverts the priority: the ask is preserved, the context is compressed.

Edge cases

• Multi-question threads need all questions extracted, not just the first.
• Questions that were already answered in-thread should be marked answered, not re-asked.
• The check needs the original thread for verification. Don't validate summaries against summaries.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_g3wZbtelCf1DfjXO1Oy3qw