Credit: sanas.ai (edited)
AI is often demoed under controlled conditions, but only live pressure tests can determine CX outcomes
Key points
AI demos often mask real-world failures that can damage brand trust and increase churn.
Shawn Zhang, Co-Founder and CTO of Sanas, discusses evaluating AI on its failures, not just its successes.
You need to evaluate these systems on their losses, not on their wins. An AI might work for 95% of calls in a demo, but the problem is the 5% where it fails. That 5% is your leaky bucket.
The polished promise of a perfect AI demo is setting CX leaders up for failure. Flawless presentations hide messy failures that cause more brand damage than the wins make up for. What matters isn’t how AI performs at its best; it’s how it handles the worst.
Shawn Zhang is the Co-Founder and CTO of Sanas, a speech AI company that transforms accents and language translation in real time. His perspective offers a reality check for CX leaders chasing scale without accounting for risk.
Churn, drip by drip: "You need to evaluate these systems on their losses, not on their wins," says Zhang. "An AI might work for 95% of calls in a demo, but the problem is the 5% where it fails." Those failures aren’t minor bugs. They’re moments that break trust, trigger churn, and stick in customers’ minds far longer than smooth interactions. "That 5% is your leaky bucket. It’s what causes customer churn and puts a permanent stain on your brand, because those are the experiences customers remember."
Chaos to case studies: Zhang warns that leaders are often seduced by polished demos that don’t reflect real-world chaos. "I’ve heard complaints about autonomous agents failing when there's background noise, or latency, or a scenario it has never encountered before," he explains. And when AI gets confused, it doesn’t just stall. It invents. "You don't want it hallucinating bad advice. That’s why you have to demand proof that it’s working in production, not just in a demo. Ask for case studies and talk to customers who are using it today," says Zhang.
I’ve heard complaints about autonomous agents failing when there's background noise, or latency, or a scenario it has never encountered before. You have to demand proof that it’s working in production, not just in a demo.
Not a zero-sum game: Zhang refutes the tired trope that AI will render human jobs obsolete. "When Uber arrived, everyone thought taxi cabs would become obsolete. But the opposite happened, and there are more taxis in operation now than before Uber because the entire market for rides expanded," he says. "I see the same thing happening in CX. Better automation will encourage more people to call in, which means more voice traffic overall. I don't think it's a zero-sum game." AI will handle the simpler inquiries, freeing up human agents for the high-stakes, nuanced cases where a premium, white-glove touch is needed.
The great equalizer: Zhang sees AI’s real value in expanding access, not replacing people. In industries like CX, accent bias has long shaped who gets hired. "It's an unfortunate truth that many businesses filtered applicants based on how thick their accent was," he says. That’s beginning to change. "We had one customer who, using our technology, was able to hire 37% more qualified applicants they previously would have passed on."
He points to other AI tools driving similar progress: flagging distressed customers, standout agents, or signs of attrition before they surface. "It can shine a spotlight into corners that might otherwise slip through the cracks, flagging a distressed customer or a high-performing agent who deserves a raise. It's about creating visibility and fairness," Zhang explains.
