Voice agent latency is the gap between the moment a caller stops speaking and the moment the agent's reply audio begins, and the practical engineering target for natural conversation is to keep that gap under a second. The gap is not one delay but four stacked ones: the system has to detect that the caller has finished, finalize the transcript, get the first token back from the language model, and get the first byte of audio back from text to speech. The stages run in sequence, so a slowdown anywhere stretches the whole turn. Long prompts, cold tool calls, and extra telephony hops stretch it further. This guide walks through each stage of the turn loop, explains barge-in and why full-duplex handling matters, covers the common causes of degradation, and lists what to listen for in a vendor demo before you buy.
What happens between a caller finishing and the agent replying?
Every reply the agent makes is the output of a turn loop with four timed stages. First, endpoint detection: the system watches the audio stream and decides the caller has finished speaking rather than just pausing. This is a genuine tradeoff, because triggering too early cuts callers off mid-thought and triggering too late adds dead air to the start of the turn. Second, ASR finalization: the speech recognizer commits its final transcript of the utterance, resolving any words it was still revising while the caller spoke. Third, model time-to-first-token: the language model reads the transcript and the call context and begins producing its response. Fourth, TTS time-to-first-byte: text to speech takes the model's output and produces the first audio the caller hears. Well-built systems overlap these stages, streaming the model's output into TTS instead of waiting for a complete answer, so speech can begin while the rest of the response is still being generated. If you are new to the underlying architecture, our post on what an AI voice agent is covers the listen, decide, act loop these stages live inside.
Want this mapped against your own call and claim volume? Book a 30-minute audit
Why is sub-second turn latency the target for natural conversation?
Because phone conversation has a rhythm, and callers notice when a system breaks it. People trade turns quickly on a call, and a reply gap that stretches well past a second reads as hesitation, confusion, or a dead line. The failure mode is not just that the call feels slow. When the agent is slow to answer, callers fill the silence: they repeat the question, add detail, or say "hello?" That new speech lands while the agent is mid-turn, so the agent now has a stale reply queued against fresh input, and the conversation degrades into the two parties talking past each other. As an engineering practice, the teams that ship production voice agents treat sub-second turns as the bar, not because a specific millisecond count is magic, but because that is the range where callers stop perceiving the gap as a delay and the interruption spiral never starts. Latency is also the clearest audible difference between a modern agent and an older phone tree; our voice agents vs IVR comparison covers the rest of that gap.
Latency is also a constraint on what an agent attempts during a live call. A lookup that takes several seconds belongs before the call or after it, not in the middle of a sentence. Flexbone plans call flows around that, deciding from recorded calls which steps happen live and which run in the background. More on AI voice agents.
See what AI can run at your facility. In a 30-minute audit we map the calls, eligibility, and follow-ups Flexbone can take off your team first.
Book an auditWhat is barge-in and why does full-duplex handling matter?
Barge-in is the moment a caller starts speaking while the agent is still talking. It happens constantly on real calls: the caller hears the answer they needed halfway through the sentence, or realizes the agent misunderstood and jumps in to correct it. Handling it requires full-duplex audio, meaning the agent listens and speaks at the same time rather than switching between modes. A full-duplex agent detects the incoming speech, stops its own playback promptly, discards the rest of its planned response, and treats what the caller just said as the new input. A half-duplex agent does none of this. It talks over the caller, finishes a paragraph nobody is listening to, and then responds to an utterance it only partially heard. Barge-in handling is also where latency and interruption interact: the faster the agent detects the interruption and yields, the less the caller has to fight it. In our deployments this is one of the first behaviors we test, because it is hard to retrofit and impossible to hide on a live call.
What makes voice agent latency degrade?
Latency rarely fails all at once; it accumulates. The usual causes, roughly in order of how often we see them: long prompts, because every instruction, policy, and transcript line added to the model's context increases the time to first token, and prompts tend to grow over the life of a deployment. Cold or slow tool calls, because a turn that requires a lookup in a scheduling system, CRM, or eligibility service inherits that system's response time, and a backend that responds lazily under load puts its delay in the middle of a live conversation. Telephony hops, because each leg between the caller's carrier, a forwarding number, a SIP provider, and the agent platform adds transit time in both directions, and audio crosses that path on every single turn. And non-streaming synthesis, where TTS waits for a full sentence or paragraph before producing audio instead of starting on the first clause. The practical consequence: latency has to be measured per turn in production, continuously, because a deployment that was fast at launch drifts as prompts and integrations grow.
What should you listen for in a demo?
A scripted happy-path demo hides latency problems, so run the demo adversarially. Interrupt the agent mid-sentence and note two things: how quickly it stops talking, and whether its next reply addresses what you said or resumes what it was saying. Go silent for several seconds after it asks a question, and see whether it waits calmly, re-prompts once, or barrels ahead. Speak while it is speaking, then stop, and check whether the transcript survived the overlap. Ask a question that forces a real system lookup, not a canned answer, and listen for how it covers the wait; a brief acknowledgment is fine, dead air followed by a rushed answer is a warning sign. Finally, ask the vendor how they measure turn latency in production and what they do when it drifts. A vendor who cannot answer that has not operated these systems at volume. The behaviors in this list are hard to fake in a live call, which is exactly why they are worth testing.
How does Flexbone engineer for latency?
We treat turn latency as a production metric, not a launch checkbox. Our agents stream at every stage that can stream, keep prompts under active review so they do not grow unbounded, and time each turn of each production call so drift shows up in the data before it shows up in caller behavior. Barge-in, silence handling, and transfer behavior are tested against recorded real-world call patterns before an agent takes live traffic. If you want to hear what this sounds like on the kinds of calls your team handles, start with our AI voice agents overview, which covers the full architecture and the deployment model behind it.