agent's reply classifier marked 'please call me' as negative - the dialer never got the callback task
For sales-outreach (SDR) agents: Classify replies into actions, not sentiments. The only classes that matter are the ones that change what the agent does: opt-out, out-of-office, bounce, meeting booked, positive, and needs-human. Everything else is 'continue sequence'. Optimize for recall on the classes that stop sends.
TL;DR
Classify replies into actions, not sentiments. The only classes that matter are the ones that change what the agent does: opt-out, out-of-office, bounce, meeting booked, positive, and needs-human. Everything else is 'continue sequence'. Optimize for recall on the classes that stop sends.
The query
agent's reply classifier marked 'please call me' as negative - the dialer never got the callback taskUse this when
- the agent follows up after opt-outs, pauses on auto-replies, or mishandles bounces
- reply routing depends on the agent understanding what the human meant
Not for
- template rendering bugs (see the template-token skill)
- suppression list sync (see the suppression skill)
Steps
Step 1: List the misclassified replies from the incident
Pull the actual reply texts the agent got wrong and label them by hand. You need the real examples: 'not right now, try Q3' is not positive, an OOO is not a conversation, a bounce notice is not a human.
**Expected: a labeled set of real replies, including every failure from the incident.
Step 2: Define the action classes explicitly
Write down the classes as actions: STOP (opt-out), PAUSE-7D (OOO), SUPPRESS (bounce), HANDOFF (meeting booked / positive), HUMAN (angry / complex), CONTINUE (everything else). If a class does not change the agent's behavior, it is not a class.
**Expected: a short class list where each class maps to exactly one agent action.
Step 3: Bias for recall on send-stopping classes
Missing an opt-out or a bounce is far worse than pausing on a false positive. Tune thresholds so opt-out, bounce, and OOO detection err toward catching too much, and put the ambiguous cases in front of a human.
**Expected: test set shows near-perfect recall on opt-out/bounce/OOO classes.
Step 4: Handle the structural cases with rules, not the model
Out-of-office and bounce notices have machine-detectable structure: auto-submitted headers, mailer-daemon senders, 'out of office' subjects. Catch these with deterministic rules before the classifier ever sees them.
**Expected: OOO and bounce notices are filtered by rules with ~100 percent precision.
Step 5: Sample and review classifications weekly
Pull a random sample of classified replies every week and check them by hand. Reply language drifts; a classifier that was right in January rots by June without review.
**Expected: a weekly review habit; drift is caught in weeks, not quarters.
Why it happens
Reply handling fails because 'positive/negative/neutral' sentiment does not map to actions. The agent needs to know 'does the sequence stop', and sentiment labels cannot answer that. Structural replies (OOO, bounces) need rules; only the genuinely ambiguous ones need a model.
Edge cases
- 'Call me Friday' without a time is a handoff, not a booking; do not invent the time.
- Threaded replies where the lead already booked need dedupe against the calendar, not just classification.
- Auto-replies in other languages need the same rule coverage; do not assume English.
Finding this skill
This skill was written for sales-outreach (SDR) agents hitting the failure above. Search Vectle for the exact error text or symptom to find it and related skills.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstINB-wR9266vvPRyfGbM1w