Customer service AI can look like a success while customers are unhappy

A Twilio blog post published November 4, 2026 warns that a common way to score customer service AI can hide unhappy customers. The score is called containment rate. It is the share of conversations the AI handles without passing them to a person. A customer who gets frustrated and hangs up still counts as handled.

Why containment misleads

Containment says nothing about whether the customer got what they needed. Someone who gets a clear answer counts. So does someone who gives up, or who tries again tomorrow on a different channel. The post says that if a company pushes the number hard enough, it can rise simply because a person is harder to reach. Author Jesse Sumrak argues for measuring outcomes instead.

The six metrics

The article covers six measures to track for both AI and human agents. Four are named plainly:

• Satisfaction (CSAT): a score from a short survey after the conversation.
• Sentiment: how the customer felt, including people who skip the survey.
• Handle time: talk, hold and wrap-up time on a conversation.
• Abandonment: customers who leave before an agent answers, or partway through an AI conversation.

The other two show whether a conversation was truly a success or a customer giving up, and what to automate next and where the AI falls short.

Keep AI and humans separate

The post says to measure satisfaction separately for AI-only and human-handled conversations. Otherwise a strong human team can hide a weak AI. Handle time needs care too. Human handle time can rise as AI takes the routine contacts, so longer human calls may just mean the easy ones are gone. Sentiment helps because customers who leave an AI chat frustrated rarely stop to rate it.

What to do

Decide what counts as resolved or satisfied before building a dashboard around it. The post then shows how to track each measure with Twilio tools such as Flex Insights and Conversation Intelligence. Some of those features are in public beta, which means they are still being tested. The post comes from a company that sells these tools, but the core advice does not depend on any one product.