How to actually measure AI agent containment (and why your vendor’s number is wrong)
“Deflection” and “containment” are not the same thing — and most dashboards report the flattering one. A practical method for measuring the number that matters.
If you only track one support-AI metric, track the right one. The trouble is that the metric most dashboards show you — “deflection” — is often the flattering one, and it can hide exactly the failures you most need to see.
Deflection vs. containment vs. resolution
- Deflection (as many vendors define it): the customer didn’t open a ticket. This counts give-ups and abandonment as wins.
- Containment: the conversation was handled end-to-end by the AI with no human involvement.
- Resolution: the customer’s underlying problem was actually solved. This is the one that correlates with CSAT and repeat contacts.
You want to measure containment and resolution, and treat any “deflection” figure that isn’t one of those with suspicion.
A practical method
- Pull a representative sample of conversations — at least a few weeks, spanning your busy and quiet periods.
- Label the endings honestly: resolved by AI, escalated to human, or abandoned. Abandoned is not deflected.
- Segment by intent. A single blended number hides the truth; a per-intent breakdown shows you where the agent is strong and where it’s quietly failing.
- Check for false resolutions: conversations the system marked “handled” where the customer came back within 48 hours on the same issue.
- Compare against your vendor’s number. The delta is your credibility risk — and your opportunity.
Rule of thumb. If your measured containment and your reported deflection differ by more than a few points, your dashboard is optimising for the wrong definition — and someone senior is about to ask why.
What to do with the number
Once you have an honest, per-intent containment number, prioritisation is easy: fix the highest-volume failing intents first. That’s the core of our Deflection Audit — measure honestly, then remediate in order of impact.
Want this measured on your own data?
The 30-Day Deflection Audit turns everything above into your numbers and a costed roadmap.