The latency budget is the product

Text agents are forgiving about time. A user waiting on a chat reply will tolerate several seconds without concluding anything is wrong. Voice has no such allowance. Human conversational turn-taking operates on a gap measured in a few hundred milliseconds, and when that gap stretches, the listener does not think the system is thinking. They think it did not hear them, and they speak again.

That is the whole design problem. Everything else in a voice agent is negotiable. The latency budget is not, and it has to be allocated before the architecture is chosen rather than measured after.

Spend the budget deliberately

The round trip contains endpointing, transcription, whatever retrieval and tool calls the turn requires, generation, and speech synthesis. Every one of those has to fit inside the gap together, which means the architecture questions are all about what can be removed from the critical path.

• Stream everything that can stream. Transcription while the user is still speaking, generation into synthesis token by token, audio out before the sentence is finished.
• Move tool calls off the critical path. Pre-fetch the likely context at call start rather than mid-turn, and keep a single synchronous lookup as the hard ceiling.
• Accept a smaller model for the conversational layer and reserve the larger one for the turns that genuinely need it.
• Budget for the slowest dependency, not the average one. A CRM lookup that usually takes two hundred milliseconds and sometimes takes three seconds will define the user's impression of the system.

In voice, a pause is not the system thinking. It is the system appearing not to have heard, and the user talks over it.

The uncanny failures are worse than the obvious ones

A voice agent that is clearly a machine is easy to interact with, because the caller calibrates immediately. One that is nearly human sets an expectation it will then break, and the break is jarring. Interruptions that are not handled, a cheerful tone delivering bad news, a perfect voice that cannot understand a postcode, a response that begins before the caller has finished the sentence. Each of these is a small failure that reads as unsettling rather than merely broken.

The practical answers are unglamorous. Handle barge-in properly so the agent stops the moment the caller speaks. Use filler and acknowledgement only where a human would. Disclose that it is an automated system at the start, which costs nothing and removes the entire category of betrayal when the caller works it out later.

Design the exit before the entrance

Every voice deployment needs a fast, unconditional path to a human, triggered by request, by repeated misrecognition, or by detected frustration. And the handoff has to carry context, because transferring a caller who then has to repeat everything is worse than not having automated the call. The transfer is the part of the system callers remember, and it is usually the part built last.