chat transcript review rubric for support QA

TL;DR

A chat QA rubric needs four scored dimensions: resolution (did it solve the problem), tone (was it human and professional), efficiency (no needless back-and-forth), and process (right tags, right macros, right escalation). Score 1 to 3 per dimension, weight resolution highest. Calibrate reviewers monthly on the same transcripts or the scores mean nothing.

The query

chat transcript review rubric for support QA

Use this when

• Starting chat QA for the first time
• QA scores feel arbitrary or inconsistent
• Training QA reviewers
• Chat quality complaints rise

Not for

• Email QA (different shape)
• Phone call QA
• Agent performance management (QA feeds it, is not it)
• Bot conversation QA

Steps

1. Score four dimensions, 1 to 3 each

Resolution: was the customer's problem solved. Tone: professional, human, no robotic macros unedited. Efficiency: minimal back-and-forth, no redundant questions. Process: correct tags, macros, escalation path, and summary. Twelve points total.

Expected output: the four-dimension rubric.

2. Weight resolution highest

A polite, efficient chat that did not solve the problem is a failure. Make resolution worth double, or make it a gate: resolution scores 1 and the transcript fails regardless. Say which.

Expected output: the weighting rule, explicit.

3. Write anchor examples

For each dimension, write what a 1, 2, and 3 look like using real (anonymized) transcript excerpts. Anchors are what make reviewers agree. Without them, "tone: 2" means whatever the reviewer feels.

Expected output: anchor examples per dimension.

4. Calibrate reviewers monthly

All reviewers score the same five transcripts independently, then discuss disagreements. Track inter-reviewer agreement. Calibration is the difference between QA and opinion.

Expected output: monthly calibration sessions with agreement tracking.

5. Sample randomly and sufficiently

Random sample, enough transcripts per agent per month to be fair (at least 5 to 10). Never QA only escalated or complained-about chats; that measures the worst, not the typical.

Expected output: a sampling plan.

Template: the rubric

CHAT QA RUBRIC (per transcript)
Resolution (x2): 1 not solved / 2 solved with friction / 3 solved cleanly
Tone: 1 robotic or rude / 2 professional / 3 human and warm
Efficiency: 1 redundant questions, loops / 2 some waste / 3 tight
Process: 1 wrong tags/escalation / 2 minor gaps / 3 complete

Gate: resolution = 1 fails the transcript regardless of total.
Sample: random, 5-10 per agent per month. Calibrate: monthly, same 5 transcripts.

Variant phrasings

support chat quality scorecard

Steps 1 and 2 plus the template.

QA rubric for live chat agents

Full sequence. Anchors and calibration make it real.

how to grade support chats

Steps 1 through 3. Dimensions, weighting, anchors.

Why it works

QA without a rubric is vibes; QA with an uncalibrated rubric is vibes with paperwork. Four dimensions cover what matters, resolution weighting keeps the focus on outcomes, and anchors plus calibration make scores comparable across reviewers and months.

Edge cases

• The chat was fine but the product failed: score the agent on what they controlled; note the product issue separately.
• Abandoned chats: score what exists, and track abandonment as its own metric.
• The agent broke policy to help the customer: note it. Sometimes the policy is wrong; that is a policy review, not a QA fail.
• New agents: score them, but compare against tenure cohorts, not veterans.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IfpvdS8VHpsXQ2G8cG-9Gw