Voice

Voice agents are conversational AI systems you talk to instead of type at — phone support lines, restaurant booking bots, in-app voice assistants. Evaluating them means evaluating a spoken conversation, which has strictly more to it than a text one:

• What was said — the transcript, which regular multi-turn metrics like TurnRelevancyMetric can judge.
• How it sounded — the actual audio on both sides of the call.
• When it happened — response latency and turn-taking behavior.

deepeval treats voice as a first-class modality, and it changes how conversations are produced, not how they are evaluated: the pipeline you already know — goldens in, test cases out, metrics on top — is untouched. A simulated voice conversation produces a standard ConversationalTestCase whose Turns carry audio and timing alongside the transcript, so your existing LLM-as-a-judge metrics keep working while the audio and timing data is preserved for voice-specific analysis.

This page explains the concepts behind deepeval's voice support — speech models, transports, connectors, voice data on test cases, and interruptions. To actually run voice simulations, see Voice Mode on the ConversationSimulator.

Why Voice Is Different

A text chatbot is a function: you call it with a message and it returns one. A deployed voice agent is a live call: audio streams in both directions over a real transport, the agent runs its own speech recognition and speech synthesis internally, and "when a turn ends" is a judgment call based on silence and timing rather than a return statement.

This has two consequences for evaluation:

1. You need speech models of your own. To simulate a user, deepeval must speak messages to your agent (text-to-speech) and listen to its replies (speech-to-text). These sit outside your agent — they are the simulated caller's mouth and ears.
2. You need a transport integration. There is no universal API for "send this audio to my agent"; agents are reachable over different protocols depending on where they're deployed. deepeval abstracts this behind connectors.

How Voice Evals Work

Evaluating a voice agent follows the same loop as evaluating a text chatbot — a simulated user talks to your application until the conversation ends, and the result is scored. The voice version inserts two speech steps into that loop and swaps the function call for a live connection:

1. Generate a user message. The simulator model role-plays the user from the ConversationalGolden's scenario and persona — exactly as in a text simulation.
2. Speak it (voice only). The TTS model synthesizes the message into audio: this is the simulated caller's voice.
3. Play it to your agent and record the reply (voice only). The connector streams the audio over the live transport, waits for your agent to finish speaking, and captures the reply audio along with latency_ms.
4. Transcribe the reply (voice only). The STT model turns the agent's audio into the assistant turn's content.
5. Decide whether to continue. Stopping logic checks the conversation against the golden's expected_outcome — exactly as in text.
6. Score the conversation. The finished ConversationalTestCase — transcript plus audio and timing — is evaluated with the same multi-turn metrics you'd use on a text conversation.

Voice Metrics

Voice metrics follow the same three-part model as the test case:

• What was said remains the responsibility of existing conversational metrics such as TurnRelevancyMetric, GoalAccuracyMetric, and ToolUseMetric.
• How it sounded is evaluated by VoiceNaturalnessMetric, SpeechIntelligibilityMetric, VoiceConsistencyMetric, and AudioIntegrityMetric.
• When it happened is evaluated by TurnTakingNaturalnessMetric and AgentResponsivenessMetric.

VoiceReliabilityMetric is the optional operational summary. It combines the responsiveness and audio-integrity checks, but a critical failure such as missing agent audio or a complete failure to respond always forces its score to zero. Minor checks cannot average away a catastrophic failure.

All voice quality metrics return a score from 0 to 1, where higher is better. Measurements such as latency, loudness, clipping rate, pause duration, and speaking rate appear in metric reasons and breakdowns as diagnostic evidence. They are not universal quality scores: whether a pause or response delay sounds natural depends on what was happening in the conversation.

The metrics use different parts of the same ConversationalTestCase:

• VoiceNaturalnessMetric, SpeechIntelligibilityMetric, and AudioIntegrityMetric analyze each assistant Turn.audio.
• VoiceConsistencyMetric compares assistant audio across multiple turns.
• TurnTakingNaturalnessMetric reconstructs the call timeline from each Turn.audio.start_time and Audio.duration, using Turn.role to identify the speaker.
• AgentResponsivenessMetric examines the ordered transcript and audio turns for missing responses and reprompts.

An audio clip without start_time can still be evaluated for how it sounded. It cannot be used to infer silence or overlap honestly, so timing metrics skip test cases whose audio has no call-relative placement.

Speech Models

Text-to-speech (TTS) and speech-to-text (STT) are separate model families from LLMs, with largely separate providers. Because they are genuinely different model types, deepeval gives them their own base classes — DeepEvalBaseTTS and DeepEvalBaseSTT — rather than bolting synthesize/transcribe methods onto DeepEvalBaseLLM. In a voice simulation the two play opposite roles, and their quality matters asymmetrically.

Text-to-Speech (TTS)

A TTS model turns text into spoken audio. In a voice simulation, the TTS model is the simulated user's voice: every user message the simulator generates is synthesized to speech before it reaches your agent.

Some LLM providers offer TTS (OpenAI, Gemini), while dedicated vendors like ElevenLabs, Cartesia, and Inworld lead on naturalness and voice variety. For simulation purposes the bar is lower than for production voice products — the synthesis needs to be clear enough that your agent's own speech recognition isn't the bottleneck, since unnatural or garbled speech skews the whole simulation.

Speech-to-Text (STT)

An STT model turns spoken audio into text. In a voice simulation, the STT model is how deepeval hears your agent: its transcription of each spoken reply becomes the Turn.content that your metrics judge.

This makes STT the more quality-critical of the two — transcription accuracy directly bounds evaluation quality, and it decides voice-specific measurements like word error rate (WER). It's also why supporting industry-standard STT providers (Deepgram, AssemblyAI, OpenAI's Whisper family) matters more than TTS breadth.

Transports

There is no single way to reach a deployed voice agent. In practice, agents are reachable over a handful of transports:

• SIP / PSTN (telephony) — the universal path. Any agent behind a phone number or SIP URI can be called, regardless of vendor: Vapi, Retell, Bland, LiveKit deployments, ElevenLabs agents with phone numbers.
• WebRTC — real-time peer-to-peer media. Used by LiveKit rooms, Pipecat, and the web-call modes of Vapi and Retell. Lowest latency, but requires a WebRTC client stack.
• WebSocket — raw audio frames over a socket. Used by ElevenLabs conversational agents and many custom in-house agents.
• REST "create call" APIs — most managed platforms have one, but it only initiates the session; the audio itself still flows over one of the transports above.

deepeval makes the transport explicit with the VoiceProtocol enum. Every connector class declares which protocol it speaks:

class VoiceProtocol(Enum):
    WEBRTC = "webrtc"        # LiveKit rooms, Pipecat, Vapi/Retell web calls
    WEBSOCKET = "websocket"  # raw-audio WS APIs (ElevenLabs ConvAI, custom agents)
    SIP = "sip"              # PSTN / telephony (Twilio et al.)
    CALLBACK = "callback"    # in-process Python callable, no transport

Two design decisions are worth calling out:

• One protocol maps to many connectors. LiveKit, Pipecat, and Vapi web calls are all WEBRTC; ElevenLabs and a custom in-house agent can both be WEBSOCKET. The protocol describes the transport, not the vendor.
• Timing semantics are defined per protocol, not per connector. How end-of-turn is detected and what latency_ms measures follows from the transport's characteristics, so latencies are comparable across connectors that share a protocol — but not across different protocols.

CALLBACK is the exception that proves the rule: it is deepeval-specific and carries no network transport at all — the "agent" is an in-process Python callable.

Connectors

A connector is deepeval's adapter between the simulator and a live audio session. It holds the call and, by default, drives it one full exchange at a time:

exchange_turn() is step 3 in How Voice Evals Work. connect() and disconnect() bracket the whole conversation — one live call per conversation:

One live call per connector is also why voice simulations run sequentially — concurrent conversations would interleave audio on the same session. The simulator maps each exchange onto a normal Turn on the ConversationalTestCase — you only construct the transport reply type yourself when wrapping an in-process agent; see Callback.

Detecting the end of a turn

The hardest part of a voice connector is knowing when the agent has finished speaking — audio streams don't come with return statements. Connectors use a turn engine that combines signals:

• Silence detection: a window of quiet after speech marks the end of the reply. Crucially this is silence since the last speech, not silence totalled across the turn, so an agent that pauses repeatedly is not eventually cut off by the sum of its pauses.
• Platform events: some protocols emit explicit turn-complete messages, which take precedence.
• Timeouts: a hard cap so an unresponsive agent can't hang the simulation.

Both thresholds come from the connector's turn_detection preset — "eager", "balanced", or "patient" — rather than being set individually. Their meaning is fixed per protocol, which is what keeps latency_ms comparable within a protocol.

Voice Data on Test Cases

Voice data lives on the same test case classes you already use, so nothing downstream has to change. A voice conversation is still a ConversationalTestCase made of Turns — see multi-turn test cases for the full structure of the Turn class — with a few voice fields added on top:

class Turn:
    role: Literal["user", "assistant"]
    content: str
    # Voice
    audio: Optional[Audio] = None
    latency_ms: Optional[float] = None
    interrupted: Optional[bool] = None
    ...

• Turn.audio holds an Audio object for that turn: the synthesized user speech on user turns, the agent's reply on assistant turns. Clip length lives on Audio.duration (seconds), not on the turn.
• Turn.latency_ms records how long the agent took to start speaking after the user's audio was sent (assistant turns only). This is wait time, not how long the reply lasted.
• Turn.interrupted is True when a user barge-in cut this assistant reply short; left None when interruptions weren't exercised (half-duplex) or the turn finished normally.

Because the transcript still lives in each Turn.content, every multi-turn metric works on voice conversations unchanged — the audio and timing fields are additional signal, not a parallel format.

Audio Data Model

Here's the data model of the Audio class in deepeval:

class Audio:
    dataBase64: Optional[str] = None
    mimeType: Optional[str] = None
    url: Optional[str] = None
    sampleRate: Optional[int] = None
    encoding: Optional[str] = None
    duration: Optional[float] = None
    start_time: Optional[float] = None

Construct an Audio in exactly one of two ways:

from deepeval.test_case import Audio

# From a local or remote file — mimeType, filename, and bytes are handled for you
recording = Audio(url="./agent-reply.wav")

# From raw bytes (e.g. TTS or connector output) — encodes into dataBase64 for you
recording = Audio.from_bytes(wav_bytes, mimeType="audio/wav", sampleRate=24000)

Audio.from_bytes(...) is the supported in-memory constructor: pass raw bytes plus mimeType, and it stores them as dataBase64 under the hood. You generally should not pass dataBase64= yourself.

There are SEVEN fields on an Audio:

• [Optional] url: a string that is a local file path or an http(s):// URL. When set, mimeType, filename, and (for local files) dataBase64 are derived for you. Defaulted to None.
• [Optional] dataBase64: a string of base64-encoded audio bytes stored on the object. Set automatically by Audio.from_bytes(...) or when loading a local url; not something you normally pass in. Defaulted to None.
• [Optional] mimeType: a string specifying the audio MIME type, such as "audio/wav", "audio/mpeg", "audio/opus", "audio/aac", "audio/flac", or "audio/pcm". Guessed from the file extension when constructing from url (falling back to "audio/wav"). Required for Audio.from_bytes(...). Defaulted to None.
• [Optional] sampleRate: an integer sample rate in Hz (e.g. 24000). Metadata only — set it when you know it. Defaulted to None.
• [Optional] encoding: a string container/codec label (e.g. "wav"). Metadata only. Defaulted to None.
• [Optional] duration: a number representing the length of this audio clip in seconds. Metadata only — this is how long the speech is, not how long the agent took to reply (Turn.latency_ms). Defaulted to None.
• [Optional] start_time: the number of seconds from the beginning of the call to the first frame of this clip. Voice simulations populate it from the live monotonic call clock. Together with duration, it lets metrics reconstruct real silence and overlap without duplicating a full-call recording. Defaulted to None for manually constructed or legacy audio.

When constructing from url, two read-only properties are derived for you: local (whether the URL points to a local file) and filename (the basename of the path). They can't be passed to the constructor.

To read the raw bytes back out — for example to compute audio measurements or write files to disk — call recording.get_bytes().

Interruptions

A live voice call is almost always bidirectional on the wire: audio can flow to the agent and from the agent at the same time. What varies is how the simulation uses that pipe.

From the simulated caller's point of view:

• Uplink — speech going out to the agent (what the agent hears).
• Downlink — speech coming back from the agent (what you record and transcribe).

That distinction matters more than how audio is packetized. User speech is typically sent as a sequence of short frames either way; the important choice is whether the simulator waits for the agent before doing anything else.

Half-duplex vs duplex

• Half-duplex — one side at a time. The simulated user speaks, then the simulator waits until the agent has finished before continuing. Overlap and barge-in are not exercised. This is the default, and it is what exchange_turn() does.
• Duplex — both directions can be active together. The simulator can listen to the agent while (or after) starting to speak, cut its own uplink short, and react when both sides talk at once. That is what makes interruptions (barge-in) evaluable.

So "duplex" here is not a second kind of connector. The same live session can be driven either way: as a strict exchange of full turns, or as concurrent uplink and downlink with interruption behavior on top.

When interruptions are enabled, the simulator stops using exchange_turn() and instead drives the same connector with:

What an interruption is

An interruption is simply overlap: uplink and downlink active at the same time.

• The user interrupts when the simulated caller starts speaking while the agent is still talking. On the resulting assistant Turn, interrupted is set to True.
• The agent "interrupts" (or keeps the floor) when agent speech continues — or resumes — while the user is speaking. The simulator notices that on the downlink and can stop the user's uplink so the call can recover, the same awkward way a phone conversation does when both people talk and then both pause.

Whether to barge in at all is a trait of the simulated caller (how aggressive they are), which is why it is configured on their persona. The concepts that matter on the call itself are only: uplink, downlink, and whether the control loop allows them to overlap.

Simulating Voice Conversations

The ConversationSimulator puts all of this together: pass a VoiceConfig (connector + optional speech models) instead of a model_callback, and every simulated conversation becomes a voice call.

from deepeval.voice import VoiceConfig, ElevenLabsConnector
from deepeval.simulator import ConversationSimulator

simulator = ConversationSimulator(
    voice_config=VoiceConfig(connector=ElevenLabsConnector(agent_id="your-agent-id")),
)

See Voice Mode for configuration and custom speech models, Voice Connectors for the connector catalog, Personas for who does the calling, and Interruptions for barge-in.

FAQs