Fair AI Support Pilot Scorecard Calculator
Interactive weighted scorecard for AI support pilots: 8 dimensions, pass/fail hard-stop gates, live recommendation, and a copy-paste readout.
Interactive scorecard
Fair AI support pilot scorecard
Score each dimension 0–10. Weights follow the RTCI framework. Hard-stop gates override the weighted total.
Weighted total
0
out of 100
Hard-stop gates
TBD
6 unevaluated
Recommendation
INCOMPLETE
0/8 scored
Confidence
Low
based on evidence
INCOMPLETE
Finish all 8 scores and 6 gates to get a recommendation.
Scoring tip: At 80% of baseline award 6/10; at 100%+ award 10/10.
Scoring tip: A >3 point drop is a yellow flag; >5 points is a red flag. Score on trend.
Scoring tip: Wrong/unacceptable must stay below 2% to score above 5/10.
Pass/fail gates (hard stops)
If any gate fails, the pilot stops regardless of weighted score.
Data and privacy compliance
No training on customer data without contract terms; passes security review; auditable access logs.
Critical error rate
Any response causing harm, policy violation, or data exposure. More than one confirmed error is a hard stop.
Customer satisfaction floor
AI CSAT more than 10 points below human baseline for 2 consecutive weeks.
Reopen or escalation flood
AI reopen rate exceeds 150% of human baseline for 2 consecutive weeks.
Agent acceptance
>30% of agents report the tool makes their job harder, or attrition spikes.
Vendor transparency
Refuses raw logs, confidence scores, or root causes more than once after formal request.
Interpretation: >80 strong candidate · 65–80 conditional · <65 stop
How this is calculated
Each of the 8 dimensions is scored 0–10 by the pilot review committee and multiplied by its RTCI weight. Six pass/fail hard-stop gates override the weighted score. The tool also loads the worked Fin example from the article and exports a committee-ready readout.
Frequently asked
Can I use this for a vendor other than Intercom Fin?
Yes. The dimensions and weights are vendor-agnostic. Adjust the targets and fields to match your baseline, ticket mix, and industry.
Why do hard-stop gates override the weighted score?
Because a 95% weighted total is meaningless if the bot leaked data, gave dangerous instructions, or the vendor refuses to share logs. Gates exist so catastrophic failures cannot be averaged away.
What does the confidence score mean?
A rough signal based on how many dimensions have evidence entered. It is not a statistical measure. Use it as a prompt to fill in missing notes before a formal readout.