All free tools

Fair AI Support Pilot Scorecard Calculator

Interactive weighted scorecard for AI support pilots: 8 dimensions, pass/fail hard-stop gates, live recommendation, and a copy-paste readout.

Interactive scorecard

Fair AI support pilot scorecard

Score each dimension 0–10. Weights follow the RTCI framework. Hard-stop gates override the weighted total.

Weighted total

0

out of 100

Hard-stop gates

TBD

6 unevaluated

Recommendation

INCOMPLETE

0/8 scored

Confidence

Low

based on evidence

INCOMPLETE

Finish all 8 scores and 6 gates to get a recommendation.

Scoring tip: At 80% of baseline award 6/10; at 100%+ award 10/10.

Scoring tip: A >3 point drop is a yellow flag; >5 points is a red flag. Score on trend.

Scoring tip: Wrong/unacceptable must stay below 2% to score above 5/10.

Pass/fail gates (hard stops)

If any gate fails, the pilot stops regardless of weighted score.

Data and privacy compliance

No training on customer data without contract terms; passes security review; auditable access logs.

Critical error rate

Any response causing harm, policy violation, or data exposure. More than one confirmed error is a hard stop.

Customer satisfaction floor

AI CSAT more than 10 points below human baseline for 2 consecutive weeks.

Reopen or escalation flood

AI reopen rate exceeds 150% of human baseline for 2 consecutive weeks.

Agent acceptance

>30% of agents report the tool makes their job harder, or attrition spikes.

Vendor transparency

Refuses raw logs, confidence scores, or root causes more than once after formal request.

Interpretation: >80 strong candidate · 65–80 conditional · <65 stop

How this is calculated

Each of the 8 dimensions is scored 0–10 by the pilot review committee and multiplied by its RTCI weight. Six pass/fail hard-stop gates override the weighted score. The tool also loads the worked Fin example from the article and exports a committee-ready readout.

Frequently asked

Can I use this for a vendor other than Intercom Fin?

Yes. The dimensions and weights are vendor-agnostic. Adjust the targets and fields to match your baseline, ticket mix, and industry.

Why do hard-stop gates override the weighted score?

Because a 95% weighted total is meaningless if the bot leaked data, gave dangerous instructions, or the vendor refuses to share logs. Gates exist so catastrophic failures cannot be averaged away.

What does the confidence score mean?

A rough signal based on how many dimensions have evidence entered. It is not a statistical measure. Use it as a prompt to fill in missing notes before a formal readout.