All posts

Published Nov 20, 2026 · Updated Nov 20, 2026 · 17 min read

Customer Service Quality Assurance: A QA Scorecard for Human and AI Conversations (2026)

One QA program that grades human agents and the AI agent on the same scorecard: nine weighted criteria with pass/fail definitions, five auto-fail checks for false resolutions, invented policy, missed handoffs, unverified actions and tone after escalation, the sample-size math, calibration with an agreement-rate example, and a cadence by team size.

Checklist card: Resolved for real, Policy is real, Verified first, Handoff on time, Tone after escalation.
The five checks that apply to every conversation, whoever answered it. A conversation that fails any one scores zero.

Customer service quality assurance (QA) means reviewing a random sample of support conversations against a written scorecard, so you know whether answers were correct, resolutions were real and customers were treated well. Score human agents and the AI agent on the same scorecard: intent, accuracy, real resolution, verified actions, handoff, communication and tone, plus five auto-fail checks. Sample about 200 conversations a month for each group you report on. That gives a 95% margin of error of ±6.9 points on a pass rate in the worst case; 400 gives ±4.9.

This guide gives the scorecard as a table and as a copyable template, the five AI failure checks, the sample-size math, how to stratify the sample, a calibration session with an agreement-rate example, a false-resolution audit, a cadence by team size, and how to turn failures into coaching and AI configuration changes. The weights, thresholds and cadences are RTCI recommendations, not industry standards.

Why grade human and AI conversations on one scorecard?

Most teams now have two quality numbers that cannot be compared: a QA score for agents from reviewers, and a resolution or containment figure for the AI agent from the vendor's dashboard. The customer sees one support team. If the AI closes a refund question with the wrong return window, that is the same failure as an agent doing it, and it should cost the same points.

One scorecard gives you pass rates per criterion that line up across both groups. You can then see, for example, that accuracy fails cluster in AI-resolved conversations while handoff fails cluster in human-only ones, and fix each where it starts. Seven criteria worth 95 points are shared. Each group gets one 5-point criterion of its own, so both still total 100.

QA is not CSAT. A survey records the opinion of customers who chose to answer; QA records whether the conversation met your standard, judged by a reviewer who read it. Run both, tag both with the same fields, and read them side by side. The survey side is covered in customer service survey questions.

The QA scorecard for human and AI conversations

Each criterion is pass or fail. Binary calls are faster to make and easier to calibrate than a 1 to 5 rating, because two reviewers only have to agree on which side of a written line the conversation falls.

#CriterionWeightPassFailApplies toAuto-fail
1Intent identified10Works on the customer's actual request, including every part of a multi-part messageAnswers a different question or drops part of the requestBothNone
2Accurate information20Every policy, price, date and step matches the source of truth that was live at the timeAny statement that is outdated, wrong or not in the sourceBothAF2
3Resolution matches status15The status (resolved, pending, escalated) matches what happened; if resolved, the customer has what they asked for or a confirmed next stepMarked resolved with the question unanswered, the customer gone mid-flow, or the fix waiting on a step nobody tookBothAF1
4Actions verified and in policy15Refunds, cancellations and account changes happen only after the required identity check, within limits, and appear in the system logAction taken without the check, outside limits, or claimed as done with no log entryBoth; N/A if no actionAF3, AF5
5Escalation and handoff15Escalates when a trigger fires (asked for a person, out of scope, two failed attempts, a listed sensitive topic) and not otherwise; the handoff note carries intent, steps tried and verification statusRequired escalation missed or late, needless escalation, or the next person has to ask againBothAF4
6Clear, complete communication10Plain language, every part answered, next step and timing statedJargon, missing steps, no next step, or contradicts itselfBothNone
7Tone, including after escalation10Matches the customer's state; after frustration or a request for a person, acknowledges it and stops pushing self-serviceScripted cheer after a complaint, repeated deflection after a request for a person, blame or sarcasmBothNone
8Ownership and follow-through5Internal note written, promised follow-up scheduled, ticket fields correctA promise with no task or note, or wrong fieldsHuman onlyNone
9Scope and disclosure5Stays inside its configured topics and identifies itself as automated where your policy requiresAnswers outside its approved scope or presents itself as a personAI onlyNone

The auto-fail items override the weighted score. If any one occurs, the conversation scores 0 and is logged under its code, because no amount of good tone makes up for an invented refund policy.

CodeAuto-failApplies toEvidence to check
AF1Marked resolved when the issue was not resolvedBothFinal customer message, 72-hour repeat contact, the customer's survey answer if any
AF2States a policy, price, entitlement or exception that does not existBothThe help article or policy version live on the conversation date
AF3Account or money action without the required verificationBothAction log timestamps against the verification event
AF4Customer asked for a person, or a mandatory trigger fired, and no handoff was offeredBothTranscript search for "agent", "person", "human", "representative"; escalation log
AF5Personal data shown or sent to an unverified partyBothMessages sent before verification completed

Scoring rule. Conversation score = weight of criteria passed ÷ weight of criteria that apply × 100. Criterion 4 drops out when no action was taken, and each group only carries its own 5-point criterion.

Worked example (assumed conversation). An AI agent answers an order-status question. No action was taken, so criterion 4 drops out: applicable weight is 100 − 15 = 85. The answer is accurate and correctly resolved, but it gives no delivery timing, so criterion 6 fails. Score = (85 − 10) ÷ 85 × 100 = 88.2. Had the agent also quoted a returns policy that does not exist, AF2 would set the score to 0.

Report three numbers, not one: the average conversation score, the share of conversations with any auto-fail, and the pass rate of each criterion. The average hides the failures that matter; the per-criterion pass rates tell you what to fix.

Copy the scorecard template

Paste this into a doc, spreadsheet or your QA tool's scorecard builder. Keep the version number in every review so a definition change shows up in the trend.

CUSTOMER SERVICE QA SCORECARD: HUMAN + AI (v1)
Score = weight passed / weight applicable x 100
Any auto-fail = 0 for the conversation

CRITERIA (pass/fail) | WEIGHT | APPLIES TO
1. Intent identified | 10 | Both
2. Accurate information | 20 | Both
3. Resolution matches status | 15 | Both
4. Actions verified and in policy | 15 | Both (N/A if no action)
5. Escalation and handoff | 15 | Both
6. Clear, complete communication | 10 | Both
7. Tone, including after escalation | 10 | Both
8. Ownership and follow-through | 5 | Human only
9. Scope and disclosure | 5 | AI only

AUTO-FAIL (score 0, log the code)
AF1 Marked resolved, issue not resolved
AF2 Policy, price or entitlement that does not exist
AF3 Account or money action without verification
AF4 Asked for a person / trigger fired, no handoff
AF5 Personal data to an unverified party

LOG PER REVIEW
conversation_id, scorecard_version, reviewer,
resolution_type (ai_resolved / ai_then_human / human_only),
channel, intent, sample (random / targeted),
criteria failed, auto-fail code, score, note

The resolution_type, channel and intent fields come from the helpdesk record, not from the reviewer's judgment. They are the same fields recommended for post-ticket surveys, so QA and CSAT results can be joined on conversation_id.

The five AI failure checks

Each auto-fail applies to people as well, but AI agents fail them in ways a transcript read alone can miss. These are the checks to run on every AI conversation in the sample.

CheckWhat it looks like in an AI conversationEvidence beyond the transcript
False "resolved" statusThe bot says it is glad to have helped and the conversation closes after the customer goes quiet, or the customer replies "ok" to an answer that did not fitRepeat contact from the same customer within 72 hours; survey answer to "Did the answer solve your problem?"
Invented policyA confident return window, fee waiver or eligibility rule that is not in any help articleThe policy or article version live on the conversation date
Missed handoffThe customer asks for a person and the bot asks them to rephrase, or offers more articlesEscalation log; number of turns between the request and the transfer
Action without verification"I've cancelled your order" before the identity step, or with no matching entry in the order systemAction and verification timestamps from the system log
Tone after escalationAfter a complaint or a request for a person, the bot keeps the upbeat script, repeats the same suggestion or ends with a sales promptMessages sent after the first escalation signal

Invented policy has a public precedent. In February 2024 a British Columbia tribunal ordered Air Canada to pay a customer damages of "just over $650" after its website chatbot suggested bereavement fares could be claimed retroactively, Global News reported on February 15, 2024. Air Canada had argued it could not be held liable for the chatbot's information; the tribunal disagreed. A QA reviewer comparing that answer with the live fare policy would have logged it as AF2.

Two of the checks need system data. Ask your AI vendor for an export with the action log and escalation events per conversation before the pilot starts; the pilot scorecard calculator treats missing logs as a hard-stop gate for the same reason.

How many conversations should you sample?

A QA pass rate is a proportion, so its margin of error follows the standard formula for a simple random sample at 95% confidence. All figures in this section are RTCI calculations, not survey results.

Margin of error E = 1.96 × √(p × (1 − p) ÷ n), where p is the pass rate as a decimal and n the number of conversations reviewed.

Sample needed n = 1.96² × p × (1 − p) ÷ E².

When you do not know the pass rate yet, use p = 0.5, which gives the widest margin. The second column shows a team whose pass rate turns out to be 90%, as an assumed example.

Conversations reviewed (n)Margin at p = 0.5 (worst case)Margin at p = 0.9 (assumed)
100±9.8 pts±5.9 pts
200±6.9 pts±4.2 pts
400±4.9 pts±2.9 pts

Worked example, n = 200, p = 0.5. 0.5 × 0.5 ÷ 200 = 0.00125. √0.00125 = 0.0354. 1.96 × 0.0354 = 0.0693, so ±6.9 points. Using that worst-case margin, a measured pass rate of 72% puts the true rate between about 65% and 79%.

To read a pass rate within ±5 points in the worst case, n = 3.8416 × 0.25 ÷ 0.0025 = 384.2, so 385 conversations. If the whole population is small, apply the finite population correction n ÷ (1 + (n − 1) ÷ N). With 2,000 conversations a month: 384.16 ÷ (1 + 383.16 ÷ 2,000) = 322.4, so 323 reviews.

Per-agent numbers are where small samples mislead. Four reviews a week is about 17 a month, which at p = 0.5 carries a margin of ±23.8 points. Even a quarter's 52 reviews is ±13.6. Use per-agent reviews to find coaching examples, and use team-level and criterion-level pass rates to make decisions. Kaizo's guide to QA sampling (July 2026, updated October 2026) makes a similar point: it describes the common practice as "often two or three tickets per agent per week" and says a sample for coaching or reviews has to be far larger.

Rare failures need a different sum. The chance that a random sample contains at least one conversation with a failure that occurs at rate r is 1 − (1 − r)^n. At r = 2% (assumed), 100 reviews catch at least one with 86.7% probability and 200 with 98.2%. To be 95% sure of seeing at least one, n = ln(0.05) ÷ ln(1 − r): 149 reviews at 2%, 299 at 1%. If an auto-fail matters at those rates, either sample more or search the full population for it, as the false-resolution audit below does.

Horizontal bar chart of the 95% margin of error on a QA pass rate by conversations reviewed: ±13.9 points at 50, ±9.8 at 100, ±6.9 at 200 (highlighted), ±4.9 at 400 and ±3.5 at 800, with a dashed line at ±5 points.
Margin of error on a QA pass rate at 95% confidence, worst case p = 0.5: ±13.9 points at 50 conversations, ±6.9 at 200 and ±4.9 at 400.

How to stratify the QA sample

A single random sample of all conversations will be dominated by whatever is most common, usually AI-resolved chat. Split the population into strata, sample randomly within each, and set a floor so small groups still get enough reviews to read.

  • By resolution path. ai_resolved, ai_then_human and human_only, from the helpdesk record. This is the split every AI-versus-human comparison depends on.
  • By channel. Chat, email, messaging and voice fail in different ways; voice needs a transcript or recording before it can be scored at all.
  • By intent. Cover your highest-volume intents, plus every high-risk intent (refunds, cancellations, account access, complaints, anything legal or safety-related) at a fixed minimum regardless of volume.

Keep random and targeted samples apart. Pulling every escalation or every low CSAT score is useful for finding problems, but it is not representative. Kaizo's guide puts it as "Keep the two apart when reporting." The sample field in the template records which is which.

Worked example (assumed volumes). A team handles 10,000 conversations a month: 5,000 AI-resolved, 2,000 AI then human, 3,000 human only. It reviews 400: 200 AI-resolved, 100 AI then human, 100 human only, so the two smaller groups get a floor instead of their proportional 80 and 120. Pass rates come back at 80%, 70% and 90%.

StratumPopulation shareReviewedPass rateWeighted contribution
AI-resolved50%20080%0.50 × 80 = 40.0
AI then human20%10070%0.20 × 70 = 14.0
Human only30%10090%0.30 × 90 = 27.0
All conversations100%40081.0%

The unweighted pass rate of the 400 reviews is (160 + 70 + 90) ÷ 400 = 80%. The population-weighted rate is 81%. Report the weighted figure for the whole operation and each stratum on its own; never average the stratum rates without the population weights.

How to run a QA calibration session

Calibration checks that two reviewers would score the same conversation the same way. Without it, a change in a QA score can come from who reviewed the conversations rather than from the conversations.

  1. Pick the set. Zendesk's QA calibration guide (Riley Young, last updated September 20, 2024) suggests starting with five conversations and adjusting. Include at least one AI-resolved, one handoff and one human-only conversation, and one borderline auto-fail.
  2. Score blind. Each reviewer scores alone, criterion by criterion. Zendesk calls this a blind session and contrasts it with a team session, where the group scores together; only the blind version measures disagreement.
  3. Compute agreement. Per criterion and overall, as in the example below.
  4. Discuss only the disagreements. The QA owner makes the call and rewrites the pass or fail definition with the conversation as an example.
  5. Publish the change. Bump the scorecard version and tell agents what changed. Zendesk's guide lists keeping agents in the loop and updating quality standards as the follow-up steps.

On frequency, Zendesk's guide says "For most teams, monthly sessions are the perfect amount," and that more reviewers means calibrating more often. If you use automated QA, treat the grader as one more reviewer in the session. Richpanel's guide to AI support QA (August 4, 2026) recommends calibrating an LLM grader against a human-scored set of 50 to 100 conversations.

Worked example (assumed scores). Reviewers A and B each score 10 conversations on the seven shared criteria: 70 pass/fail calls each. They agree on 61 calls: 49 where both passed and 12 where both failed. A passed 52 calls in total and B passed 55.

StepCalculationResult
Percent agreement (Po)61 ÷ 7087.1%
Chance agreement (Pe)(52 ÷ 70 × 55 ÷ 70) + (18 ÷ 70 × 15 ÷ 70)63.9%
Cohen's kappa(0.871 − 0.639) ÷ (1 − 0.639)0.64

Raw agreement looks high, but most calls are passes, so two reviewers would agree often by chance. Kappa removes that chance agreement. In her review of the statistic, McHugh (Biochemia Medica, October 2012) notes that "many texts recommend 80% agreement as the minimum acceptable," and reads any kappa below 0.60 as inadequate agreement. This pair clears both, narrowly.

RTCI's rule: compute both numbers per criterion, rewrite any criterion below 80% agreement or 0.60 kappa, and require full agreement on auto-fail calls before QA scores feed coaching plans or performance reviews. Ten conversations is a small set, so read one session's kappa as a prompt for discussion and track it across sessions.

How to audit AI-resolved conversations for false resolutions

A false resolution is a conversation the AI marked resolved where the customer's problem did not end. Some show up as repeat contacts; others are customers who gave up. The audit finds the first group from data and estimates the second from a sample.

  1. Take the population. Every conversation the AI platform marked resolved with no human involvement in the period.
  2. Join repeat contacts. Flag any conversation where the same customer contacted you again about the same issue within 72 hours. These are presumed false resolutions. The method and the join are set out in how to measure AI resolution rate.
  3. Sample the rest. Draw a random sample from the conversations with no repeat contact and read each one with the scorecard, checking AF1 first.
  4. Classify each sampled conversation. Verified resolved; unresolved and abandoned; wrong answer accepted by the customer; actually escalated but miscoded.
  5. Estimate the total. Repeat contacts plus the sampled false rate × the rest of the population, with the margin shown.

Worked example (assumed counts). The AI marked 5,000 conversations resolved in a month. 400 had a repeat contact within 72 hours. Of the other 4,600, QA reviewed a random 200 and found 18 unresolved (9%).

StepCalculationResult
Margin on the sampled rate1.96 × √(0.09 × 0.91 ÷ 200)±4.0 pts
Estimated silent false resolutions0.09 × 4,600414
Estimated false resolutions, total400 + 414814 (16.3% of 5,000)
Range400 + (0.050 to 0.130) × 4,600about 632 to 996
Verified resolutions5,000 − 814about 4,186

The vendor dashboard says 5,000 resolutions; the audit supports about 4,186, with a plausible range of roughly 4,004 to 4,368. Which number you report depends on whether you count at first contact or end to end, a choice covered in first-contact vs end-to-end resolution.

If you pay per resolution, the audit is also a billing check. The 72-hour verified resolution clauses give you contract language for disputing resolutions that reopen, and the pilot scorecard calculator is where the verified number goes when you score a pilot.

QA cadence and owners by team size

These cadences are RTCI recommendations sized to the sampling math above, not an industry standard. Adjust the review counts to your reviewers' time, but keep the monthly totals per reported group near 200 where you can.

Small (under 10 agents)Mid (10 to 50 agents)Large (50+ agents)
QA ownerSupport lead, part timeDedicated QA lead plus rotating senior agentsQA team; a named owner for AI quality in CX operations
Human reviews3 per agent per week3 to 5 per agent per weekPer-agent quota plus stratified sample by queue
AI reviews50 random AI-resolved a week50 to 100 a week, stratified by intent200+ a month per channel, stratified by intent
After an AI change25 conversations from the first days50 from the first days50 per changed intent
CalibrationMonthly, lead plus one senior agentMonthly, all reviewersEvery two weeks per reviewer group, monthly across groups
False-resolution auditMonthlyMonthlyWeekly join, monthly sample
ReportingMonthly to the teamWeekly to support leadershipWeekly, with AI and human pass rates per criterion

The "after an AI change" row follows a point Richpanel makes in the guide cited above: AI errors cluster by cause, so one bad knowledge edit can repeat the same wrong answer across many customers, and it recommends keying review to deploys rather than the calendar.

Using QA results in coaching and AI configuration

A failed criterion tells you where to make the fix. For people, the fix is coaching. For the AI agent, it is a configuration change, which then needs a check that it worked.

Failed criterionHuman agent: coaching actionAI agent: configuration fixCheck the fix
1 IntentPractice restating multi-part requests before answeringAdd the missed phrasing to intent examples; split intents that overlapRerun the failed conversations
2 Accuracy / AF2Show where the current policy lives; retire saved replies that are out of dateCorrect or add the source article; restrict answers on that topic to retrieved contentAsk the original question again in a test conversation
3 Resolution / AF1Confirm the outcome with the customer before closingChange close conditions so silence is not a resolution; add a confirmation stepFalse-resolution audit next month
4 Actions / AF3, AF5Walk through the verification steps for that actionRequire the verification tool call before the action is availableTest the action with an unverified user
5 Handoff / AF4Review escalation triggers and handoff note formatAdd the missed trigger; transfer on the first request for a personCount turns from request to transfer
6 CommunicationRewrite the reply together using one model answerAdd a next-step and timing rule to answer instructionsSpot-check 10 conversations
7 Tone after escalationRole-play the first reply after a complaintChange the handoff message; stop suggesting articles once escalation startsRead messages after the escalation signal

Coach one criterion at a time, using the agent's own conversation, and track that criterion's pass rate over the following weeks rather than the total score. Keep every AI conversation that failed as a regression set, and rerun it after each knowledge or prompt change so a fixed failure stays fixed.

Handoff failures often sit between the two. If the AI sends the right conversations but the agent still has to ask the customer to repeat everything, the fix is the handoff packet, covered in training the team on AI handoff workflows.

Frequently asked questions

What is customer service quality assurance?

Customer service quality assurance (QA) is a regular review of a random sample of support conversations against a written scorecard. A reviewer checks whether the agent understood the request, gave correct information, actually resolved the issue, followed verification and escalation rules, and communicated well. With AI agents answering customers, the same scorecard should cover the AI's conversations too.

What should a customer service QA scorecard include?

RTCI's scorecard has nine pass/fail criteria weighted to 100: intent (10), accurate information (20), resolution matches status (15), actions verified and in policy (15), escalation and handoff (15), communication (10) and tone (10) for everyone, plus ownership (5) for humans or scope and disclosure (5) for AI. Five auto-fail items set the conversation score to zero.

How many conversations should customer service QA review?

For a pass rate you report, about 200 conversations per group per month gives a 95% margin of error of ±6.9 points in the worst case, and about 400 gives ±4.9. The formula is 1.96 × √(p(1 − p) ÷ n). Per-agent samples of a few conversations a week are too small to rank agents and are best used for coaching examples.

How do you QA AI agent conversations?

Score them on the same criteria as human conversations and add five auto-fail checks: marked resolved when the issue was not, a stated policy that does not exist, an account or money action without verification, a missed handoff, and personal data shown to an unverified person. Pull a separate random sample of AI-resolved conversations and join it to 72-hour repeat contacts to find false resolutions.

What is a good QA calibration score?

McHugh (Biochemia Medica, 2012) notes that many texts recommend 80% agreement as the minimum acceptable, and reads a Cohen's kappa below 0.60 as inadequate agreement. RTCI suggests both thresholds per criterion, and full agreement on auto-fail calls before scores are used in coaching or performance reviews.

How often should QA calibration sessions happen?

Zendesk's QA calibration guide says monthly sessions suit most teams, and that more reviewers means calibrating more often. RTCI adds one trigger: calibrate again after any scorecard change and after any change to the AI agent's knowledge, prompts or actions, because the definitions of pass and fail shift with them.

Change log

  1. Published.

Read next

Picked for this article: the pieces it cites and the ones that answer the next question buyers usually ask.

All 62 posts

Most related

Metrics10 minNov 20, 2026

How to Measure AI Resolution Rate Internally

Vendor dashboards report resolution rates that no two vendors calculate the same way. This is a practical, vendor-agnostic methodology to measure your own AI resolution rate using your own ticket data, built on RTCI's 72-hour verified resolution standard.

Read the piece