Voice agents: why the pauses matter more than the words
A text assistant can take a few seconds and nobody minds. A voice agent can’t. In a phone call, silence is part of the conversation, and a long gap makes people think the line has dropped. So with voice, how fast it answers matters as much as what it says.
Where the time goes
The classic setup chains three steps. Speech is turned into text, a language model decides what to say, and text is turned back into speech. Each step adds delay, and they add up. Some newer systems handle speech directly, which can cut steps, but the same questions apply.
Figure 1 · The voice pipeline
Four things that decide how it feels
Figure 2 · What to test
| Issue | What goes wrong | What helps |
|---|---|---|
| Knowing when to reply | It cuts in early, or waits too long | Tune how it detects the end of a turn |
| Slow responses | Long silences | Smaller or faster models for simple turns, and streaming the reply as it’s generated |
| Interruptions | It keeps talking when the caller speaks | Stop speaking the moment the caller starts |
| Mistakes | Wrong name, number or date | Read important details back to the caller |
Know when to hand over
A good voice agent knows its limits. If the caller is upset, if the same question has failed twice, or if the request is outside what it should do, it should pass the call to a person with the context it has gathered. Designing that handover well is half the work, as we said about review points in the human-in-the-loop note.
Voice agents are one of the project types we build. If you have a use case in mind, tell us about it.