All posts

Published Nov 20, 2026 · Updated Nov 20, 2026 · 9 min read

Hot Take: Judge Your AI Agent by How It Hands Off, Not by Its Resolution Rate

RTCI's opinion: the conversations an AI agent cannot solve decide whether customers trust it, and they are where billing and measurement games hide. Four public failures, a five-point handoff standard, five metrics, contract clauses and what would change our mind.

Flow diagram: Customer → AI agent → Human agent → Resolved, with Human agent highlighted and a dotted branch from AI agent to Loop.
The step that decides trust is the one after the AI runs out of answers: getting the customer to a person, with the context, without a loop.

RTCI's view: judge an AI support agent first by what it does when it cannot solve the problem, and treat its resolution rate as the second number you check. Resolution rate is the figure vendors sell and buyers compare, but customers decide whether to trust AI support in the conversations it fails: the loop with no way out, the invented policy, the agent who asks them to start again. A bad handoff turns a 20% miss into a lost customer.

The handoff is also where the measurement and billing games are played. A customer who comes back the next day gets counted as resolved; a conversation gets billed and then passed to a human. This is an opinion piece, argued from RTCI's index data and four public cases, and it ends with the evidence that would prove it wrong.

Why is resolution rate the wrong headline metric?

First, the headline is rarely the real number. Across the 20 platforms in RTCI's index (November 2026), the median vendor claim is 74% and the median realistic resolution rate is 58.3%, a median gap of 17 points per platform. Why vendor-claimed resolution rates don't match reality shows how definitions and denominators produce that gap.

Second, even an honest resolution rate describes the easy majority. It tells you how often the AI finished the job. It says nothing about the conversations it did not finish, 41.7% of them at the index's median realistic rate (100 − 58.3), which are the ones with an angry customer, a refund, a dispute or a policy question on them.

Third, the bill often ignores the line between resolved and handed off. Nine of the 20 platforms in RTCI's index list a per-conversation or per-session price. For five it is the model: Kore.ai (sessions plus seats), Robylon (credits per conversation), Kustomer AI ($0.75 per engaged conversation), Freshdesk Freddy ($0.49 per AI session) and Tidio Lyro ($0.50 to $0.65 per AI conversation). For four it is an option: Decagon, Ada CX, Crescendo and Salesforce Agentforce (per conversation for existing customers only). Intercom Fin prices per outcome, and its outcomes include Procedure handoffs, billed at $0.99 like a resolution, per the Intercom Fin review.

Worked example. Assumptions: 10,000 Freddy AI sessions a month at the published $0.49, ignoring the 500 free sessions and agent seats, and Freddy's realistic resolution midpoint of 56.5% (RTCI range 52–62%). The AI bill is 10,000 × $0.49 = $4,900. It resolves 5,650 sessions, so 4,350 end with a human or with nobody, and those cost 4,350 × $0.49 = $2,131.50, or 43.5% of the bill. Cost per resolution is $4,900 ÷ 5,650 = $0.87. None of that is unfair if the price is known, but it means the vendor is paid the same whether the miss is handled well or badly.

Per-resolution billing has the mirror-image problem: everything rides on what counts as resolved. HubSpot Breeze publishes a definition, a resolution with no handoff within 72 hours. Where a contract counts any bot-ended chat instead, a customer who gives up and emails the next day is billed as a success. Illustration only: if a vendor counts 7,000 of 10,000 conversations as resolved and 10% of those customers return within 72 hours, the real rate is (7,000 − 700) ÷ 10,000 = 63%, not 70%, and 700 of the billed resolutions were handoffs that took a day longer.

What happens when the edge fails?

Four public cases, each opened by RTCI on November 20, 2026. None is a resolution-rate problem. Each is a conversation where the right move was to stop answering, or to let the customer out of the loop.

Air Canada, February 2024. British Columbia's Civil Resolution Tribunal ordered Air Canada to pay a customer just over $650 in damages after its website chatbot suggested bereavement fares could be claimed after travel, which the airline's policy did not allow, Global News reported on February 15, 2024. Air Canada argued it could not be held liable for the chatbot's information. The tribunal disagreed. The bot answered a fare-policy question it should have grounded in the policy page or passed to an agent.

Cursor, April 2025. Cursor's support bot, "Sam", told a user who kept being logged out that the product was designed for one device per subscription. No such policy existed. The user posted on Reddit, other users said they were cancelling, and a Cursor representative corrected it about three hours later, after the story had spread, CX Today reported on April 21, 2025. A bot with a rule to escalate account-access problems would have produced a ticket instead of a policy.

Klarna, May 2025. CEO Sebastian Siemiatkowski told Bloomberg that cost had been "a too predominant evaluation factor" in how Klarna organised service, that the result was lower quality, and that customers should know "there will be always a human if you want", as CX Dive reported on May 9, 2025. The share of chats Klarna's AI handled kept rising after that, to 80% in 2025, per its annual report. The Klarna case study shows Klarna never published the re-contact or escalation rate that would show what happened to the rest.

The CFPB, June 2023. The US Consumer Financial Protection Bureau's issue spotlight Chatbots in consumer finance estimated that about 37% of US adults, roughly 98 million people, used a bank's chatbot in 2022. It described customers stuck in loops with no "offramp to a human customer service representative" and bots that struggle to recognise disputes. Its press release of June 6, 2023 warned that a poorly deployed chatbot can lead to violations of federal consumer financial law, and said institutions should avoid chatbots as their primary service channel when the bot cannot meet customers' needs.

Put together: a tribunal held the company to what its bot said, users spread a bad answer within hours, a CEO who led with AI says the human option has to stay, and a federal regulator treats the missing exit as a compliance risk.

What does a good AI-to-human handoff look like?

RTCI's five-point standard. A vendor demo should show all five on your own tickets, not on a script.

  1. The customer can leave at any turn. Asking for a person gets a person, or a ticket with a stated reply time, on the next turn. No "let me try one more thing" loop and no hidden button.
  2. The AI escalates on its own when it should. Low confidence, a second failed attempt, disputes, payments, cancellations, legal or regulatory rights and account access go to a human by rule. The CFPB's point about missed disputes is the reason the list is written down.
  3. Context travels. The human gets the transcript, the detected intent, what the AI already tried, and any verification already done, so the customer never repeats the story. Train the team on AI handoff workflows lists the packet fields.
  4. The customer is told the truth about what happens next. That a person is taking over, roughly when, and what happens out of hours. A handed-off conversation is never closed as resolved.
  5. The AI does not invent to avoid a handoff. When approved content is silent, it says so and escalates. Air Canada and Cursor are both failures of this rule.

How should you measure AI handoffs?

Five numbers, reported monthly by intent and separately for AI-only, hybrid and human-only conversations:

MetricFormulaWhat a bad result looks like
Handoff rateConversations passed to a human ÷ conversations the AI handled, with a reason code on eachLow rate alongside high re-contact: the bot is holding customers it cannot help
Time to humanTime from handoff (or the customer's request) to the first human reply; median and 90th percentileA long tail after hours, or requests for a person that never reach a queue
Context-carried rateSampled handoffs where the customer did not have to restate known facts ÷ handoffs sampledAgents opening with "how can I help?"
Post-handoff resolutionHanded-off conversations resolved by a human with no re-contact in 72 hours ÷ handed-off conversationsBelow your human-only rate for the same intents
72-hour re-contactAI-resolved conversations where the customer came back on the same issue within 72 hours ÷ AI-resolved conversationsRising as the AI's resolution rate rises

The last row is the check on the other four: a vendor can lower its handoff rate by refusing to escalate, but those customers come back. The 72-hour rule explains the window. Escalation management has the trigger matrix and handoff note template, reducing escalations without adding headcount covers how to cut avoidable handoffs without cutting needed ones, and customer service quality assurance has a scorecard for reviewing sampled handoffs. Put the five numbers next to resolution rate in the fair AI support pilot scorecard calculator before you pick a vendor.

What should the contract say about handoffs?

  • Resolution definition. Resolved means no human handoff and no re-contact on the same issue within 72 hours. A conversation that ends in a handoff is never a resolution.
  • Handoff billing. Handed-off conversations are not billed, or are billed at a stated rate below the resolution rate. Re-contacts inside 72 hours are credited.
  • Itemised invoices. Every billed event is tagged resolved, handed off, abandoned or other, with a conversation ID you can audit.
  • Exit on request. The agent routes a customer who asks for a person on the next turn, and the vendor reports how often that rule fired.
  • Escalation triggers. A written list of intents that always go to a human (disputes, payments, cancellations, legal rights, account access), which you can change without a change order.
  • Context packet. The fields passed to your helpdesk on every handoff, tested before go-live.
  • Monthly reporting. The five metrics above, by intent, as raw data you can recompute.
  • Remedy. If 72-hour re-contact or post-handoff resolution misses an agreed level for two months running, a fee credit or the right to exit.

The contract-to-go-live checklist covers the rest of the agreement.

What would change our mind?

This view is wrong if handoff quality adds nothing that resolution rate does not already tell you. The evidence that would show it: conversation-level data, from a vendor or a buyer, across several deployments, where 72-hour re-contact and CSAT on handed-off conversations track the AI's resolution rate so closely that ranking vendors by one ranks them by the other. If that holds, resolution rate is a sufficient headline and the extra metrics are overhead. RTCI has not seen such data published, and every case above points the other way, but we would update the advice if it appeared.

How RTCI calculates these numbers

  • Realistic resolution rate. Each vendor’s published claim is discounted by one fixed rule for every vendor, with an extra discount when the figure measures deflection rather than resolution.
  • Cost per resolution. Per-session and per-conversation prices are divided by that realistic rate, because failed attempts are billed too. Mandatory platform fees are spread over 2,000 to 4,000 resolutions a month.
  • Prices. Every price links to the vendor’s own page as of the date shown. Costs marked ~ or est. are RTCI estimates for vendors that publish no per-unit price.

Full methodology · Editorial policy · Index data (CSV)

Frequently asked questions

What is a good AI to human handoff?

One where the customer can ask for a person at any point and get one, the AI escalates on its own when it is unsure or when money, rights or a dispute is involved, the human receives the transcript and what the AI already tried, the customer is told what happens next and how long it takes, and the AI never invents an answer to avoid escalating. RTCI's five-point standard sets these out.

Should you pay for conversations the AI escalates to a human?

RTCI's view is no, or at a lower rate stated in the contract. Nine of the 20 platforms in RTCI's index list a per-conversation or per-session price, alone or as an option, and Intercom Fin bills Procedure handoffs as $0.99 outcomes. Under those models a conversation that ends with a human is still billed, so ask for invoices that break billed events down by type.

How fast should an AI agent hand off to a human?

When the customer asks for a person, on the next turn, with no retry loop. For other triggers, set the rule by risk: immediately for disputes, payments and legal rights; after a stated number of failed attempts for routine questions. Then measure time to human from the handoff to the first human reply, and report the median and the 90th percentile, since the slow tail is what customers remember.

Is a low AI handoff rate a good sign?

Only if 72-hour re-contact on AI-resolved conversations is also low. A bot can cut its handoff rate by refusing to escalate, closing chats or answering anyway, and each of those shows up later as a repeat contact. Read the two numbers together, by intent.

Change log

  1. Published.

Read next

Picked for this article: the pieces it cites and the ones that answer the next question buyers usually ask.

All 69 posts

Most related

Metrics17 minNov 20, 2026

Escalation Management in Customer Service: Process, Tiers, SLAs and AI Handoffs (2026)

A working escalation management system for teams running AI and human agents together: the three kinds of escalation, a copyable nine-row escalation matrix with example targets, a step-by-step process and escalation note template, hard-coded AI handoff triggers and a handoff audit, five metrics with formulas and a worked example, de-escalation scripts, and the FCA and CFPB complaint clocks.

Read the piece