Published Nov 20, 2026 · Updated Nov 20, 2026 · 17 min read
Customer Service Quality Assurance: A QA Scorecard for Human and AI Conversations (2026)
One QA program that grades human agents and the AI agent on the same scorecard: nine weighted criteria with pass/fail definitions, five auto-fail checks for false resolutions, invented policy, missed handoffs, unverified actions and tone after escalation, the sample-size math, calibration with an agreement-rate example, and a cadence by team size.

Customer service quality assurance (QA) means reviewing a random sample of support conversations against a written scorecard, so you know whether answers were correct, resolutions were real and customers were treated well. Score human agents and the AI agent on the same scorecard: intent, accuracy, real resolution, verified actions, handoff, communication and tone, plus five auto-fail checks. Sample about 200 conversations a month for each group you report on. That gives a 95% margin of error of ±6.9 points on a pass rate in the worst case; 400 gives ±4.9.
This guide gives the scorecard as a table and as a copyable template, the five AI failure checks, the sample-size math, how to stratify the sample, a calibration session with an agreement-rate example, a false-resolution audit, a cadence by team size, and how to turn failures into coaching and AI configuration changes. The weights, thresholds and cadences are RTCI recommendations, not industry standards.
Why grade human and AI conversations on one scorecard?
Most teams now have two quality numbers that cannot be compared: a QA score for agents from reviewers, and a resolution or containment figure for the AI agent from the vendor's dashboard. The customer sees one support team. If the AI closes a refund question with the wrong return window, that is the same failure as an agent doing it, and it should cost the same points.
One scorecard gives you pass rates per criterion that line up across both groups. You can then see, for example, that accuracy fails cluster in AI-resolved conversations while handoff fails cluster in human-only ones, and fix each where it starts. Seven criteria worth 95 points are shared. Each group gets one 5-point criterion of its own, so both still total 100.
QA is not CSAT. A survey records the opinion of customers who chose to answer; QA records whether the conversation met your standard, judged by a reviewer who read it. Run both, tag both with the same fields, and read them side by side. The survey side is covered in customer service survey questions.
The QA scorecard for human and AI conversations
Each criterion is pass or fail. Binary calls are faster to make and easier to calibrate than a 1 to 5 rating, because two reviewers only have to agree on which side of a written line the conversation falls.
| # | Criterion | Weight | Pass | Fail | Applies to | Auto-fail |
|---|---|---|---|---|---|---|
| 1 | Intent identified | 10 | Works on the customer's actual request, including every part of a multi-part message | Answers a different question or drops part of the request | Both | None |
| 2 | Accurate information | 20 | Every policy, price, date and step matches the source of truth that was live at the time | Any statement that is outdated, wrong or not in the source | Both | AF2 |
| 3 | Resolution matches status | 15 | The status (resolved, pending, escalated) matches what happened; if resolved, the customer has what they asked for or a confirmed next step | Marked resolved with the question unanswered, the customer gone mid-flow, or the fix waiting on a step nobody took | Both | AF1 |
| 4 | Actions verified and in policy | 15 | Refunds, cancellations and account changes happen only after the required identity check, within limits, and appear in the system log | Action taken without the check, outside limits, or claimed as done with no log entry | Both; N/A if no action | AF3, AF5 |
| 5 | Escalation and handoff | 15 | Escalates when a trigger fires (asked for a person, out of scope, two failed attempts, a listed sensitive topic) and not otherwise; the handoff note carries intent, steps tried and verification status | Required escalation missed or late, needless escalation, or the next person has to ask again | Both | AF4 |
| 6 | Clear, complete communication | 10 | Plain language, every part answered, next step and timing stated | Jargon, missing steps, no next step, or contradicts itself | Both | None |
| 7 | Tone, including after escalation | 10 | Matches the customer's state; after frustration or a request for a person, acknowledges it and stops pushing self-service | Scripted cheer after a complaint, repeated deflection after a request for a person, blame or sarcasm | Both | None |
| 8 | Ownership and follow-through | 5 | Internal note written, promised follow-up scheduled, ticket fields correct | A promise with no task or note, or wrong fields | Human only | None |
| 9 | Scope and disclosure | 5 | Stays inside its configured topics and identifies itself as automated where your policy requires | Answers outside its approved scope or presents itself as a person | AI only | None |
The auto-fail items override the weighted score. If any one occurs, the conversation scores 0 and is logged under its code, because no amount of good tone makes up for an invented refund policy.
| Code | Auto-fail | Applies to | Evidence to check |
|---|---|---|---|
| AF1 | Marked resolved when the issue was not resolved | Both | Final customer message, 72-hour repeat contact, the customer's survey answer if any |
| AF2 | States a policy, price, entitlement or exception that does not exist | Both | The help article or policy version live on the conversation date |
| AF3 | Account or money action without the required verification | Both | Action log timestamps against the verification event |
| AF4 | Customer asked for a person, or a mandatory trigger fired, and no handoff was offered | Both | Transcript search for "agent", "person", "human", "representative"; escalation log |
| AF5 | Personal data shown or sent to an unverified party | Both | Messages sent before verification completed |
Scoring rule. Conversation score = weight of criteria passed ÷ weight of criteria that apply × 100. Criterion 4 drops out when no action was taken, and each group only carries its own 5-point criterion.
Worked example (assumed conversation). An AI agent answers an order-status question. No action was taken, so criterion 4 drops out: applicable weight is 100 − 15 = 85. The answer is accurate and correctly resolved, but it gives no delivery timing, so criterion 6 fails. Score = (85 − 10) ÷ 85 × 100 = 88.2. Had the agent also quoted a returns policy that does not exist, AF2 would set the score to 0.
Report three numbers, not one: the average conversation score, the share of conversations with any auto-fail, and the pass rate of each criterion. The average hides the failures that matter; the per-criterion pass rates tell you what to fix.
Copy the scorecard template
Paste this into a doc, spreadsheet or your QA tool's scorecard builder. Keep the version number in every review so a definition change shows up in the trend.
CUSTOMER SERVICE QA SCORECARD: HUMAN + AI (v1)
Score = weight passed / weight applicable x 100
Any auto-fail = 0 for the conversation
CRITERIA (pass/fail) | WEIGHT | APPLIES TO
1. Intent identified | 10 | Both
2. Accurate information | 20 | Both
3. Resolution matches status | 15 | Both
4. Actions verified and in policy | 15 | Both (N/A if no action)
5. Escalation and handoff | 15 | Both
6. Clear, complete communication | 10 | Both
7. Tone, including after escalation | 10 | Both
8. Ownership and follow-through | 5 | Human only
9. Scope and disclosure | 5 | AI only
AUTO-FAIL (score 0, log the code)
AF1 Marked resolved, issue not resolved
AF2 Policy, price or entitlement that does not exist
AF3 Account or money action without verification
AF4 Asked for a person / trigger fired, no handoff
AF5 Personal data to an unverified party
LOG PER REVIEW
conversation_id, scorecard_version, reviewer,
resolution_type (ai_resolved / ai_then_human / human_only),
channel, intent, sample (random / targeted),
criteria failed, auto-fail code, score, note
The resolution_type, channel and intent fields come from the helpdesk record, not from the reviewer's judgment. They are the same fields recommended for post-ticket surveys, so QA and CSAT results can be joined on conversation_id.
The five AI failure checks
Each auto-fail applies to people as well, but AI agents fail them in ways a transcript read alone can miss. These are the checks to run on every AI conversation in the sample.
| Check | What it looks like in an AI conversation | Evidence beyond the transcript |
|---|---|---|
| False "resolved" status | The bot says it is glad to have helped and the conversation closes after the customer goes quiet, or the customer replies "ok" to an answer that did not fit | Repeat contact from the same customer within 72 hours; survey answer to "Did the answer solve your problem?" |
| Invented policy | A confident return window, fee waiver or eligibility rule that is not in any help article | The policy or article version live on the conversation date |
| Missed handoff | The customer asks for a person and the bot asks them to rephrase, or offers more articles | Escalation log; number of turns between the request and the transfer |
| Action without verification | "I've cancelled your order" before the identity step, or with no matching entry in the order system | Action and verification timestamps from the system log |
| Tone after escalation | After a complaint or a request for a person, the bot keeps the upbeat script, repeats the same suggestion or ends with a sales prompt | Messages sent after the first escalation signal |
Invented policy has a public precedent. In February 2024 a British Columbia tribunal ordered Air Canada to pay a customer damages of "just over $650" after its website chatbot suggested bereavement fares could be claimed retroactively, Global News reported on February 15, 2024. Air Canada had argued it could not be held liable for the chatbot's information; the tribunal disagreed. A QA reviewer comparing that answer with the live fare policy would have logged it as AF2.
Two of the checks need system data. Ask your AI vendor for an export with the action log and escalation events per conversation before the pilot starts; the pilot scorecard calculator treats missing logs as a hard-stop gate for the same reason.
How many conversations should you sample?
A QA pass rate is a proportion, so its margin of error follows the standard formula for a simple random sample at 95% confidence. All figures in this section are RTCI calculations, not survey results.
Margin of error E = 1.96 × √(p × (1 − p) ÷ n), where p is the pass rate as a decimal and n the number of conversations reviewed.
Sample needed n = 1.96² × p × (1 − p) ÷ E².
When you do not know the pass rate yet, use p = 0.5, which gives the widest margin. The second column shows a team whose pass rate turns out to be 90%, as an assumed example.
| Conversations reviewed (n) | Margin at p = 0.5 (worst case) | Margin at p = 0.9 (assumed) |
|---|---|---|
| 100 | ±9.8 pts | ±5.9 pts |
| 200 | ±6.9 pts | ±4.2 pts |
| 400 | ±4.9 pts | ±2.9 pts |
Worked example, n = 200, p = 0.5. 0.5 × 0.5 ÷ 200 = 0.00125. √0.00125 = 0.0354. 1.96 × 0.0354 = 0.0693, so ±6.9 points. Using that worst-case margin, a measured pass rate of 72% puts the true rate between about 65% and 79%.
To read a pass rate within ±5 points in the worst case, n = 3.8416 × 0.25 ÷ 0.0025 = 384.2, so 385 conversations. If the whole population is small, apply the finite population correction n ÷ (1 + (n − 1) ÷ N). With 2,000 conversations a month: 384.16 ÷ (1 + 383.16 ÷ 2,000) = 322.4, so 323 reviews.
Per-agent numbers are where small samples mislead. Four reviews a week is about 17 a month, which at p = 0.5 carries a margin of ±23.8 points. Even a quarter's 52 reviews is ±13.6. Use per-agent reviews to find coaching examples, and use team-level and criterion-level pass rates to make decisions. Kaizo's guide to QA sampling (July 2026, updated October 2026) makes a similar point: it describes the common practice as "often two or three tickets per agent per week" and says a sample for coaching or reviews has to be far larger.
Rare failures need a different sum. The chance that a random sample contains at least one conversation with a failure that occurs at rate r is 1 − (1 − r)^n. At r = 2% (assumed), 100 reviews catch at least one with 86.7% probability and 200 with 98.2%. To be 95% sure of seeing at least one, n = ln(0.05) ÷ ln(1 − r): 149 reviews at 2%, 299 at 1%. If an auto-fail matters at those rates, either sample more or search the full population for it, as the false-resolution audit below does.

How to stratify the QA sample
A single random sample of all conversations will be dominated by whatever is most common, usually AI-resolved chat. Split the population into strata, sample randomly within each, and set a floor so small groups still get enough reviews to read.
- By resolution path. ai_resolved, ai_then_human and human_only, from the helpdesk record. This is the split every AI-versus-human comparison depends on.
- By channel. Chat, email, messaging and voice fail in different ways; voice needs a transcript or recording before it can be scored at all.
- By intent. Cover your highest-volume intents, plus every high-risk intent (refunds, cancellations, account access, complaints, anything legal or safety-related) at a fixed minimum regardless of volume.
Keep random and targeted samples apart. Pulling every escalation or every low CSAT score is useful for finding problems, but it is not representative. Kaizo's guide puts it as "Keep the two apart when reporting." The sample field in the template records which is which.
Worked example (assumed volumes). A team handles 10,000 conversations a month: 5,000 AI-resolved, 2,000 AI then human, 3,000 human only. It reviews 400: 200 AI-resolved, 100 AI then human, 100 human only, so the two smaller groups get a floor instead of their proportional 80 and 120. Pass rates come back at 80%, 70% and 90%.
| Stratum | Population share | Reviewed | Pass rate | Weighted contribution |
|---|---|---|---|---|
| AI-resolved | 50% | 200 | 80% | 0.50 × 80 = 40.0 |
| AI then human | 20% | 100 | 70% | 0.20 × 70 = 14.0 |
| Human only | 30% | 100 | 90% | 0.30 × 90 = 27.0 |
| All conversations | 100% | 400 | 81.0% |
The unweighted pass rate of the 400 reviews is (160 + 70 + 90) ÷ 400 = 80%. The population-weighted rate is 81%. Report the weighted figure for the whole operation and each stratum on its own; never average the stratum rates without the population weights.
How to run a QA calibration session
Calibration checks that two reviewers would score the same conversation the same way. Without it, a change in a QA score can come from who reviewed the conversations rather than from the conversations.
- Pick the set. Zendesk's QA calibration guide (Riley Young, last updated September 20, 2024) suggests starting with five conversations and adjusting. Include at least one AI-resolved, one handoff and one human-only conversation, and one borderline auto-fail.
- Score blind. Each reviewer scores alone, criterion by criterion. Zendesk calls this a blind session and contrasts it with a team session, where the group scores together; only the blind version measures disagreement.
- Compute agreement. Per criterion and overall, as in the example below.
- Discuss only the disagreements. The QA owner makes the call and rewrites the pass or fail definition with the conversation as an example.
- Publish the change. Bump the scorecard version and tell agents what changed. Zendesk's guide lists keeping agents in the loop and updating quality standards as the follow-up steps.
On frequency, Zendesk's guide says "For most teams, monthly sessions are the perfect amount," and that more reviewers means calibrating more often. If you use automated QA, treat the grader as one more reviewer in the session. Richpanel's guide to AI support QA (August 4, 2026) recommends calibrating an LLM grader against a human-scored set of 50 to 100 conversations.
Worked example (assumed scores). Reviewers A and B each score 10 conversations on the seven shared criteria: 70 pass/fail calls each. They agree on 61 calls: 49 where both passed and 12 where both failed. A passed 52 calls in total and B passed 55.
| Step | Calculation | Result |
|---|---|---|
| Percent agreement (Po) | 61 ÷ 70 | 87.1% |
| Chance agreement (Pe) | (52 ÷ 70 × 55 ÷ 70) + (18 ÷ 70 × 15 ÷ 70) | 63.9% |
| Cohen's kappa | (0.871 − 0.639) ÷ (1 − 0.639) | 0.64 |
Raw agreement looks high, but most calls are passes, so two reviewers would agree often by chance. Kappa removes that chance agreement. In her review of the statistic, McHugh (Biochemia Medica, October 2012) notes that "many texts recommend 80% agreement as the minimum acceptable," and reads any kappa below 0.60 as inadequate agreement. This pair clears both, narrowly.
RTCI's rule: compute both numbers per criterion, rewrite any criterion below 80% agreement or 0.60 kappa, and require full agreement on auto-fail calls before QA scores feed coaching plans or performance reviews. Ten conversations is a small set, so read one session's kappa as a prompt for discussion and track it across sessions.
How to audit AI-resolved conversations for false resolutions
A false resolution is a conversation the AI marked resolved where the customer's problem did not end. Some show up as repeat contacts; others are customers who gave up. The audit finds the first group from data and estimates the second from a sample.
- Take the population. Every conversation the AI platform marked resolved with no human involvement in the period.
- Join repeat contacts. Flag any conversation where the same customer contacted you again about the same issue within 72 hours. These are presumed false resolutions. The method and the join are set out in how to measure AI resolution rate.
- Sample the rest. Draw a random sample from the conversations with no repeat contact and read each one with the scorecard, checking AF1 first.
- Classify each sampled conversation. Verified resolved; unresolved and abandoned; wrong answer accepted by the customer; actually escalated but miscoded.
- Estimate the total. Repeat contacts plus the sampled false rate × the rest of the population, with the margin shown.
Worked example (assumed counts). The AI marked 5,000 conversations resolved in a month. 400 had a repeat contact within 72 hours. Of the other 4,600, QA reviewed a random 200 and found 18 unresolved (9%).
| Step | Calculation | Result |
|---|---|---|
| Margin on the sampled rate | 1.96 × √(0.09 × 0.91 ÷ 200) | ±4.0 pts |
| Estimated silent false resolutions | 0.09 × 4,600 | 414 |
| Estimated false resolutions, total | 400 + 414 | 814 (16.3% of 5,000) |
| Range | 400 + (0.050 to 0.130) × 4,600 | about 632 to 996 |
| Verified resolutions | 5,000 − 814 | about 4,186 |
The vendor dashboard says 5,000 resolutions; the audit supports about 4,186, with a plausible range of roughly 4,004 to 4,368. Which number you report depends on whether you count at first contact or end to end, a choice covered in first-contact vs end-to-end resolution.
If you pay per resolution, the audit is also a billing check. The 72-hour verified resolution clauses give you contract language for disputing resolutions that reopen, and the pilot scorecard calculator is where the verified number goes when you score a pilot.
QA cadence and owners by team size
These cadences are RTCI recommendations sized to the sampling math above, not an industry standard. Adjust the review counts to your reviewers' time, but keep the monthly totals per reported group near 200 where you can.
| Small (under 10 agents) | Mid (10 to 50 agents) | Large (50+ agents) | |
|---|---|---|---|
| QA owner | Support lead, part time | Dedicated QA lead plus rotating senior agents | QA team; a named owner for AI quality in CX operations |
| Human reviews | 3 per agent per week | 3 to 5 per agent per week | Per-agent quota plus stratified sample by queue |
| AI reviews | 50 random AI-resolved a week | 50 to 100 a week, stratified by intent | 200+ a month per channel, stratified by intent |
| After an AI change | 25 conversations from the first days | 50 from the first days | 50 per changed intent |
| Calibration | Monthly, lead plus one senior agent | Monthly, all reviewers | Every two weeks per reviewer group, monthly across groups |
| False-resolution audit | Monthly | Monthly | Weekly join, monthly sample |
| Reporting | Monthly to the team | Weekly to support leadership | Weekly, with AI and human pass rates per criterion |
The "after an AI change" row follows a point Richpanel makes in the guide cited above: AI errors cluster by cause, so one bad knowledge edit can repeat the same wrong answer across many customers, and it recommends keying review to deploys rather than the calendar.
Using QA results in coaching and AI configuration
A failed criterion tells you where to make the fix. For people, the fix is coaching. For the AI agent, it is a configuration change, which then needs a check that it worked.
| Failed criterion | Human agent: coaching action | AI agent: configuration fix | Check the fix |
|---|---|---|---|
| 1 Intent | Practice restating multi-part requests before answering | Add the missed phrasing to intent examples; split intents that overlap | Rerun the failed conversations |
| 2 Accuracy / AF2 | Show where the current policy lives; retire saved replies that are out of date | Correct or add the source article; restrict answers on that topic to retrieved content | Ask the original question again in a test conversation |
| 3 Resolution / AF1 | Confirm the outcome with the customer before closing | Change close conditions so silence is not a resolution; add a confirmation step | False-resolution audit next month |
| 4 Actions / AF3, AF5 | Walk through the verification steps for that action | Require the verification tool call before the action is available | Test the action with an unverified user |
| 5 Handoff / AF4 | Review escalation triggers and handoff note format | Add the missed trigger; transfer on the first request for a person | Count turns from request to transfer |
| 6 Communication | Rewrite the reply together using one model answer | Add a next-step and timing rule to answer instructions | Spot-check 10 conversations |
| 7 Tone after escalation | Role-play the first reply after a complaint | Change the handoff message; stop suggesting articles once escalation starts | Read messages after the escalation signal |
Coach one criterion at a time, using the agent's own conversation, and track that criterion's pass rate over the following weeks rather than the total score. Keep every AI conversation that failed as a regression set, and rerun it after each knowledge or prompt change so a fixed failure stays fixed.
Handoff failures often sit between the two. If the AI sends the right conversations but the agent still has to ask the customer to repeat everything, the fix is the handoff packet, covered in training the team on AI handoff workflows.
Frequently asked questions
What is customer service quality assurance?
Customer service quality assurance (QA) is a regular review of a random sample of support conversations against a written scorecard. A reviewer checks whether the agent understood the request, gave correct information, actually resolved the issue, followed verification and escalation rules, and communicated well. With AI agents answering customers, the same scorecard should cover the AI's conversations too.
What should a customer service QA scorecard include?
RTCI's scorecard has nine pass/fail criteria weighted to 100: intent (10), accurate information (20), resolution matches status (15), actions verified and in policy (15), escalation and handoff (15), communication (10) and tone (10) for everyone, plus ownership (5) for humans or scope and disclosure (5) for AI. Five auto-fail items set the conversation score to zero.
How many conversations should customer service QA review?
For a pass rate you report, about 200 conversations per group per month gives a 95% margin of error of ±6.9 points in the worst case, and about 400 gives ±4.9. The formula is 1.96 × √(p(1 − p) ÷ n). Per-agent samples of a few conversations a week are too small to rank agents and are best used for coaching examples.
How do you QA AI agent conversations?
Score them on the same criteria as human conversations and add five auto-fail checks: marked resolved when the issue was not, a stated policy that does not exist, an account or money action without verification, a missed handoff, and personal data shown to an unverified person. Pull a separate random sample of AI-resolved conversations and join it to 72-hour repeat contacts to find false resolutions.
What is a good QA calibration score?
McHugh (Biochemia Medica, 2012) notes that many texts recommend 80% agreement as the minimum acceptable, and reads a Cohen's kappa below 0.60 as inadequate agreement. RTCI suggests both thresholds per criterion, and full agreement on auto-fail calls before scores are used in coaching or performance reviews.
How often should QA calibration sessions happen?
Zendesk's QA calibration guide says monthly sessions suit most teams, and that more reviewers means calibrating more often. RTCI adds one trigger: calibrate again after any scorecard change and after any change to the AI agent's knowledge, prompts or actions, because the definitions of pass and fail shift with them.
Change log
- Published.
Read next
Keep reading
Picked for this article: the pieces it cites and the ones that answer the next question buyers usually ask.

Most related
How to Measure AI Resolution Rate Internally
Vendor dashboards report resolution rates that no two vendors calculate the same way. This is a practical, vendor-agnostic methodology to measure your own AI resolution rate using your own ticket data, built on RTCI's 72-hour verified resolution standard.

First-Contact vs End-to-End Resolution
A metric decision guide for AI support teams: when to use first-contact resolution rate vs end-to-end resolution rate, how to track both, and how deflection fits into the same dashboard.

Train the Team on AI Handoff Workflows
What a clean AI-to-human handoff packet contains, how to measure handoff quality, what vendors bill when a conversation is handed off, and how handoffs move cost per verified resolution.

72-Hour Verified Resolution SLA: Copy-Paste Contract Clauses
A buyer-side clause library for defining, measuring, auditing, crediting, and disputing 72-hour verified resolutions in AI support contracts.

Customer Service Survey Questions: 35 Customer Satisfaction Survey Questions for Services, With AI-Handled and Human Tickets
A 35-question bank for the survey that goes out when a support ticket closes: CSAT, effort, yes/no and open-text wording, separate lines for AI-handled and hand-off conversations, the hidden fields to tag, and how many responses a CSAT gap needs before it means anything.