TL;DR: A voice agent can pass every text test and still fail on a real phone call. Speech adds problems text never shows: interruptions, silence, background noise, accents, names and dates heard wrong, and callers who give three facts at once. In Sierra's τ-Voice benchmark, the same tasks that text agents completed 85% of the time dropped to 31 to 51% for voice agents in clean audio and 26 to 38% in realistic conditions (τ-Voice, 2026). Test over the real phone path, score audio and meaning separately, and check the booking or record the call was supposed to create.
Most voice agent teams start testing in text. It's fast and cheap, and it catches prompt and logic bugs. Then the agent goes live and callers start reporting problems the text suite never saw. This guide explains why that gap exists, which call scenarios expose it, and how to test voice agents so the result matches what callers actually experience.
Quick answers
Why does a voice agent pass in text but fail on the phone? Text tests skip the speech layer. On a call, the agent has to hear correctly, decide when to speak, cope with interruptions and silence, and carry corrected details through several turns. Each of those can break a task the text version handled fine.
Is the problem usually speech recognition? Not always. The τ-Voice authors found that 79 to 90% of failures came from agent behavior rather than the simulated caller or other factors (τ-Voice, 2026). Losing track of a multi-step request is as common as mishearing a word.
How does ContextQA test this? ContextQA places real calls to your agent over a phone number, Amazon Connect, WebRTC or SIP, using generated caller personas. It scores each call with LLM judges and deterministic checks, measures audio signals such as latency, interruptions and dead air, and can check the backend record with a read-only verifier (ContextQA voice agent testing).
What changes between text and a real call?
The prompt, tools and knowledge base can be identical. What changes is everything around them.
| Layer | What text testing sees | What a phone call adds | What to check |
|---|---|---|---|
| Hearing | Clean typed input | Accents, noise, line quality, misheard names and numbers | Entities captured correctly, not just pronounced well |
| Timing | Instant turns | Latency, dead air, talking over the caller | Measured response latency and silence against your policy |
| Turn-taking | One message at a time | Interruptions, barge-in, "wait" or "no" mid-sentence | The agent yields and doesn't treat "no" as approval |
| Context | Clear, single facts | Several facts in one breath, corrections, repeats | The final corrected value is the one used |
| Outcome | A reply that sounds right | A booking, update or transfer that must really happen | The persisted record matches what was agreed |
The last row is the one teams skip most. A clearly spoken confirmation with a high audio score doesn't prove that the right appointment was created (ContextQA booking outcome docs).
Which call scenarios break voice agents?
Build your suite around the situations real callers create. These seven come up again and again:
- The mid-sentence correction. The caller says "Thursday at two, actually no, make it Friday" while the agent is reading back the booking. The agent has to drop the old date and confirm the new one.
- Several facts at once. "It's for Maria Lopez, a cleaning, next Tuesday morning." All three values have to survive.
- The short interjection. The caller says "no" or "wait" during a readback. The agent must not treat it as a yes.
- Silence. The caller pauses to find a card or check a calendar. The agent shouldn't talk over them or hang up.
- Names, dates and numbers. Spelled names, relative dates such as "next Friday", times near midnight or across timezones, and policy or claim numbers.
- Noise and accents. A caller on speakerphone in a car, or with an accent your test personas never used.
- The repeated correction. The caller corrects the same value twice. The final value must win.
ContextQA's docs describe the same patterns as test cases: a calm baseline, a combined answer, a mid-readback correction, a clarification, a short interjection and a repeated correction, each with its own assertion (ContextQA interruptions and context docs).

How should you score a voice call?
Score sound and meaning separately. One broad "conversation quality" score hides the failure you need to find.
- Audio criteria answer "did it sound right?": audio quality, tone, language match, turn-taking and dead air.
- Measured signals answer "what happened in the audio?": response latency, agent interruptions, whether the agent yields when interrupted, dead air and talk ratio. ContextQA lists these as measured criteria you enable per project (ContextQA voice metrics reference).
- Meaning criteria answer "did it understand?": multi-turn context, the final corrected values and identity.
- Outcome checks answer "did the work happen?": the booking, record or transfer, read back from the system of record.
Keep them separate in reports. An agent can yield politely to an interruption and still book the old date. A wrong date pronounced clearly still fails the date check.
Want voice agents with policy, grounding and outcome checks on every call? See how ShipVoice by ContextQA verifies the outcome before a call is marked resolved.
How do you build a voice test suite that matches reality?
- Write the expected outcome first. For each scenario, define the final values and the record that should exist when the call ends.
- Start with matched text and voice cases. Run the same request both ways. If text passes and voice fails, you've narrowed the problem to the speech path or turn handling, though timing and session handling can matter too.
- Test over the real path. Call the agent the way customers do: by phone number, through your contact-center platform or over WebRTC. A web simulator can miss telephony issues.
- Use varied caller personas. ContextQA generates personas and use cases from your agent brief and knowledge base, then builds test cases with expected outputs (ContextQA voice testing docs). Add accents, speaking speeds and noisy conditions that match your real callers.
- Isolate test data. Give each success case its own booking slot or record. If one test takes the only open slot, the next one fails for a reason that has nothing to do with the agent.
- Verify the backend. Add a read-only verifier that looks up the record for this test by its ID, not the "latest booking" (ContextQA verifier docs).
- Replay failures. Listen to the recording around the failure, read the transcript, then compare with the tool calls and the stored record.
Our broader guide to voice AI testing in CI covers running these suites on every change, and IVR testing covers keypad menus.
What does a real failure look like?
Here's a typical pattern, using a made-up clinic receptionist agent:
- In text, a caller asks to move a Thursday cleaning to Friday at 10:30. The agent finds the appointment, offers slots and books Friday. The test passes.
- On the phone, the caller starts with Thursday, then interrupts the readback to say "sorry, Friday works better". The agent yields correctly and says "Friday at 10:30, you're all set."
- The verifier finds the appointment still booked for Thursday. The agent confirmed the new time aloud but sent the original date to the reschedule tool.
Every audio score on that call is green. The caller still turns up on the wrong day. Only the outcome check catches it. For more on scoring the business result, see business-outcome testing across AI agents and enterprise apps.
Bottom line
Text tests are a useful first filter, but they can't tell you how a voice agent behaves when a real person interrupts, pauses, corrects themselves or calls from a noisy street. Test over the real phone path, score audio, meaning and outcome separately, and check the system of record before you call a task complete. The gap between text and voice is large in published benchmarks, so measure your own instead of assuming it's small.
Getting a voice agent ready for real callers? Book a ContextQA demo and bring your hardest call scenario.
Sources
- τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains (arXiv, 2026)
- Sierra: τ-Voice benchmark announcement
- ContextQA docs: Voice AI testing
- ContextQA docs: Interruptions and multi-turn context
- ContextQA docs: Voice AI metrics reference
- ContextQA docs: Verify voice AI bookings and business outcomes
- ContextQA docs: Verify tool actions and business outcomes