How to Build Reliable Voice AI Products: Testing, Latency, and the Context Problem

The gap between a voice AI demo and a voice AI product

There is a moment in every voice AI project when the demo works perfectly and the team feels like the hard part is done. The transcript is clean, the response is fast, and the conversation flows naturally. Then real users arrive.

They speak with accents. They trail off mid-sentence. They reference things said three exchanges earlier. They ask questions the demo script never anticipated. And the gap between what the demo showed and what the product delivers becomes visible almost immediately.

This gap is not a bug. It is a structural consequence of how voice AI systems are built, and closing it requires a different kind of engineering rigor than most teams apply.

In a recent episode of the TechEffect podcast, we sat down with Dr. Varun Singh, Chief Product and Technology Officer at Daily and lead architect of Pipecat, the open-source orchestration framework for voice and multimodal conversational AI now used by hundreds of companies. With two decades of experience at the intersection of real-time communication and AI infrastructure, Singh offered one of the clearest frameworks we've heard for why voice AI products fail in production, and what rigorous engineering looks like at every layer of the pipeline.

Why voice AI is harder to build than it looks

The first thing to understand about production voice AI is that what appears to be a single system is almost always three systems running in sequence.

Singh describes the standard architecture: "You have a speech to text, LLM, text to speech. And these things are now called cascades because you're cascading three models."

Each model in the cascade has its own latency profile, its own accuracy characteristics, and its own failure modes. Speech-to-text systems can stumble on accented speech, non-standard vocabulary, or incomplete sentences. The LLM introduces its own latency and is sensitive to how well the transcribed text captures what the user actually meant. Text-to-speech adds a final rendering delay before anything reaches the user's ear.

In isolation, each component can test well. In combination, under real network conditions and real conversational patterns, the failure modes compound in ways that unit-level testing will not reveal.

There is also a deeper issue. The cascade architecture means that errors at one stage propagate downstream. A mis-transcription does not just produce a bad transcript—it sends a corrupted input to the LLM, which then generates a response to something the user never said. The text-to-speech system then faithfully renders that incorrect response. By the time the user hears something wrong, the root cause is two steps back and not immediately visible.

The parallelization problem

Getting the cascade to run fast enough for a natural conversation requires more than sequential optimization. Singh describes the practical target: "While you're transcribing the speech you start the LLM processing so that you can send the text so that the LLM can start to formulate an answer while I'm still speaking."

This kind of parallelization is the difference between a voice assistant that feels responsive and one that feels like it is making you wait. But it also creates new testing requirements. The system has to be tested not just for what each component produces, but for how they perform when they are all running concurrently under real load.

Network topology adds another layer of complexity. "You have your own model, I have my own model, he has his own model," Singh explains. "At some point you have these three models that you're trying to, like, work with. You want to place your bot as close to these three models, you want these three models to be close to each other as well." Geographic distribution of the models is a latency variable that most teams do not systematically test.

The two metrics that actually matter

Before any testing strategy can be designed, teams need agreement on what they are actually measuring. Singh cuts through the complexity with a framework developed across two decades of building real-time systems.

"There are lots of metrics that you can measure, but typically the thesis is there are only two things that you really care about. One is how quickly you can send video and render video on the other side—a timeliness metric. And the other is quality."

Timeliness, in the context of voice AI, is end-to-end latency: the time between when the user finishes speaking and when the first audio response arrives. Quality is the accuracy and coherence of what gets said.

These two dimensions are in constant tension. A system optimized purely for speed may generate responses before the LLM has had time to formulate a good one. A system optimized purely for quality may buffer long enough to produce better responses but at the cost of the conversational rhythm that makes the interaction feel natural.

Singh draws the contrast sharply: "What you want to avoid is a walkie-talkie situation—you say hello and then you wait for an audited time before the other party says hello because that's the contract you have."

The practical implication for testing: both dimensions need continuous instrumentation, not just at the component level but glass-to-glass, meaning from the moment the user speaks to the moment the response is audible. Optimizing one model in isolation will not reveal the combined latency of the full pipeline.

Context engineering: The problem nobody planned for

If latency is the engineering challenge that teams expect, context management is the one that surprises them.

In a single-turn interaction—one question, one answer—the LLM sees the full context of what was asked and has everything it needs to respond. As conversations extend, the problem changes fundamentally. What has been said before, what the user assumed was remembered, and what the model has actually retained begin to diverge.

Singh describes what this looks like in practice: "Multi-turn conversations also have the problem of how much of the conversation should it remember? If it's a long conversation—15, 20 turns —does it remember the beginning of the call?"

This is not a solvable problem in the way that a bug is solvable. It is a tradeoff that has to be actively managed. Token limits mean that at some point the full conversation history can no longer fit in the model's context window. Something has to be compressed, summarized, or discarded.

Singh explains the engineering response: "You had prompt engineering before, now you have context engineering. You have a million tokens, you want to make sure you use as much but not all of it. As the conversation goes longer, you want to compress it. You may also want to keep a raw transcript on the side in case you compressed differently."

The implication for testing is significant. A voice AI system cannot be validated only on short, clean conversations. It has to be tested across the full distribution of conversation length, with deliberate coverage of the edge cases where context compression decisions have the most impact on response quality.

You might be interested in: How to Test Audio Quality in AI Voice Agents

Why speaking is not the same as typing

There is a property of spoken language that makes this problem harder than it appears in text-based systems.

Singh identifies it directly: "When we talk with each other we have shared context which we rely on quite a lot. So we don't specify in great detail, you can say 'this' and 'that' and by context know what 'this' and 'that' would mean. But if you do that in chat, it doesn't have that context."

When users type into a chatbot, they tend to be more explicit. The act of writing encourages completeness. When they speak, they naturally use pronouns, references, and assumptions that only make sense given the shared context of the conversation so far. A system that handles well-formed written queries may fail badly on the abbreviated, pronoun-heavy speech that users actually produce in voice interactions.

This has direct implications for how evaluation datasets are built. If the test conversations used to validate a voice AI system are written rather than transcribed from real speech, they will systematically underrepresent the failure modes that matter most.

Multi-agent architectures and their testing implications

Single-model pipelines are giving way to multi-agent architectures,and this shift is changing both what voice AI products can do and how they need to be tested.

Singh describes the pattern that Pipecat is increasingly built to support: "You have one LLM whose main job is to just make sure the conversation keeps on track, one to make sure it doesn't veer off and has guardrails around it. You have like a triangle—the agent actually called out and got the refund policy, is going to probably say something. You want to make sure whatever it pulled is accurate."

The analogy he uses is the call center supervisor: "In the old world with the contact center you have the call center agent and then you have a supervisor eavesdropping on the conversation." The second LLM plays the same role—watching, checking, and intervening if the primary agent begins to go wrong.

This is not just an architectural choice. It is a testing multiplier. Every agent in the pipeline is a component that can behave unexpectedly. The interactions between agents introduce emergent failure modes that would not be visible by testing any individual model in isolation. A multi-agent system needs to be validated as a system, not as a collection of individually well-behaved components.

Singh is explicit about the value this structure creates for reliability:

"You could have like this kind of mixture of experts. One LLM whose main job is to just make sure that the conversation keeps on track, one to make sure it doesn't veer off and has guardrails around it. It's going to talk about refund policy. Here's what it's saying. Does this match our refund policy? Accurate information."

For teams building customer-facing voice AI, the question is not whether to implement this kind of oversight architecture—it is how to test it rigorously enough that the supervisory LLM is actually catching errors rather than rubber-stamping them.

Pipecat and the infrastructure for voice-first products

Singh's work on Pipecat is the practical expression of everything described above. Pipecat is an open-source, vendor-neutral orchestration framework designed to make the hard parts of voice AI infrastructure accessible without forcing teams to choose a specific stack.

"Pipecat is an open-source vendor-neutral orchestration framework. You're not locked in with anything. We support something like 100 services—text to speech, LLMs, speech to text—best-in-class for many in terms of latency and accuracy. We've also made sure that it's tuned to the point where these processes run in parallel as fast."

The framework handles context management primitives—keeping context, updating it, compressing it—and exposes them at a level where developers can make deliberate decisions rather than relying on defaults. For multi-LLM configurations, it provides opinionated defaults that wire the supervisor pattern without requiring teams to implement the plumbing themselves.

The design principle behind this is time to value: getting a working system into testing quickly enough that teams can discover real failure modes before they become production failures. "You can get to that aha moment faster," Singh explains. "There will be a lot more tuning still to be done, but you can get to like 70%, 80%, 60% - depending on your use case and complexity."

What this means for teams building voice AI today

The trajectory Singh describes is not speculative. The pieces are already in place.

Speech-to-text systems have become accurate enough to handle multiple languages, accents, and non-standard vocabulary. LLMs have the knowledge, multi-turn capability, and instruction-following precision to handle a wide range of use cases. Text-to-speech has crossed the threshold from robotic to natural.

What separates products that work from products that fail in production is not the quality of any individual component. It is the rigor of the engineering and testing that connects them—the latency measurement that catches pipeline slowdowns before users notice, the context management strategy that keeps long conversations coherent, the multi-agent architecture that provides a check on what the primary model says, and the evaluation coverage that extends beyond clean demo conversations to the full distribution of how real users actually speak.

Singh frames the direction simply:

"A lot of the voice AI is now going to mimic human behavior—how much knowledge it has, does it need to pull knowledge from elsewhere, how do we make sure in a long conversation that it remembers, that it doesn't get lost along the way."

For teams that want to close the gap between the demo and the product, that is the specification.

Essentially, building voice AI that works in production requires instrumented latency measurement across the full cascade pipeline, deliberate context engineering for multi-turn conversations, multi-agent oversight architectures, and evaluation strategies built on real spoken language, not just the clean inputs that perform well in development.

→ Listen to the full conversation with Dr. Varun Singh on the Tech Effect podcast

Key takeaways

Latency and quality are the only metrics that matter in real-time voice AI. Everything else is a proxy. Instrument both end-to-end—from the moment the user speaks to the moment they hear a response—not just at the individual model level.

The cascade architecture creates compounding failure modes. A mis-transcription does not just produce a bad transcript—it corrupts the LLM input, which corrupts the response. Pipeline-level testing is the only way to catch these failures before they reach users.

Context management is a first-class engineering concern. Conversations beyond 15–20 turns require deliberate strategies for what to keep, what to compress, and what to discard. Test specifically for context degradation in long conversations.

Spoken language is structurally different from typed input. Users speak in shorter, fuzzier, pronoun-heavy sentences that assume shared context. Evaluation datasets built on clean written prompts will systematically miss the failure modes that matter in production.

Multi-agent architectures multiply your testing surface. Each additional LLM in the pipeline is a component that can fail and an interaction that can produce emergent behavior. Validate the system as a whole, not just the components.

Open-source frameworks like Pipecat compress time to value. Getting into testing quickly—at 60% or 70% of the way to production quality—allows teams to discover real failure modes before they become production incidents.