How an AI phone agent listens, thinks and talks back in a blink

An AI phone agent can begin talking before it has finished writing its answer. That is one of the details in a new guide from ElevenLabs on how a voice agent works. The company says its agent starts speaking once the language model has produced "enough words and a comma." The guide is from the vendor, so its speed claims are its own.

What a voice agent is made of

The guide describes a stack of layers. They cover speech recognition, turn-taking, a large language model (the AI that writes replies), orchestration, tools, knowledge retrieval, speech synthesis, telephony and monitoring. In short, the agent turns audio into text, decides what to say, and turns the reply back into audio. Steps overlap, so the caller waits less.

Hearing the caller correctly

Speech recognition comes first, and mistakes here carry through everything after. A misheard booking reference could pull up the wrong customer record. ElevenLabs says its Scribe v2 Realtime tool transcribes in about 150 milliseconds and is trained for noise, accents, names, emails and IDs.

The company also reports the lowest word error rate of any low-latency recognition model on a 30-language benchmark called FLEURS. A separate medical version is reported at a 7.0% word error rate on the Eka Medical benchmark. These figures come from ElevenLabs, not an outside test.

Knowing when to talk

Turn-taking decides when the agent replies and when it stays quiet. The guide says a gap of about 200 milliseconds is typical between people in conversation. Reply too early and the agent cuts callers off. Wait too long and the call feels broken.

ElevenLabs says its model reads tone, rhythm and word context, not just silence. It says the signal to stop talking when a caller interrupts lands in under 200 milliseconds. It also reports a median of about 150 milliseconds from request to audible speech with Eleven v4 Turbo.

Settings a business can change

• Reply speed: Eager, Normal or Patient. Patient suits phone numbers, addresses and emails.
• Silence timeout: 1 to 30 seconds. The guide suggests 5 to 10 for casual calls.
• Filler speech, such as "Hhmmmm...yeah.", when the AI is slow to answer.
• Custom mode lets you pick the models. Realtime mode lets ElevenLabs manage them.

The guide notes that starting a reply before the caller is confirmed finished cuts delay but can raise AI costs. Part of the guide was cut off in the material provided, so the remaining layers are not covered here.