Voice Reliability

Q: Why is my score exactly 0 instead of somewhere in between?

A critical failure was detected. Check critical_failure on each half of the breakdown to see which side it came from, then read that half's events list for the specific failure.

Multi-turn

Referenceless

Chatbot

The voice reliability metric is the operational summary of a call: did the agent answer, and did its audio arrive intact? It combines agent responsiveness and audio integrity into a single score between 0 and 1.

Usage

Voice metrics evaluate calls produced by the ConversationSimulator in voice mode. Start with one ConversationalGolden, simulate one call, then hand what it returns to evaluate():

from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import VoiceReliabilityMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig


golden = ConversationalGolden(
    scenario="The user calls to cancel an appointment and reschedule it.",
    expected_outcome="The agent cancels the old appointment and books a new one.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(
        connector=ElevenLabsConnector(agent_id="your-agent-id"),
    )
)
test_cases = simulator.simulate([golden])

metric = VoiceReliabilityMetric(threshold=0.9)
evaluate(test_cases=test_cases, metrics=[metric])

There are FIVE optional parameters when creating a VoiceReliabilityMetric:

[Optional] threshold: a number representing the minimum passing score. Set it to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] include_reason: a boolean which, when set to True, includes a summary of the evaluation score. Defaulted to True.
[Optional] strict_mode: a boolean which, when set to True, requires a perfect score. Any score below 1 becomes 0, and the threshold is set to 1. Defaulted to False.
[Optional] verbose_mode: a boolean which, when set to True, includes the score breakdown in verbose metric logs. Defaulted to False.
[Optional] flaky: a boolean which, when set to True, marks the metric as flaky. Defaulted to False.

Every voice metric records what it measured on score_breakdown. Read it after a standalone measure() call, or turn on verbose_mode to have it printed during an evaluate() run:

metric.measure(test_cases[0])

print(metric.score)  # e.g. 0.94
print(metric.score_breakdown)

For VoiceReliabilityMetric the breakdown nests both halves, each with its own score, critical flag, and events, so a failure can be traced without re-running the individual metrics:

{
    "critical_failure": False,
    "responsiveness": {
        "score": 1.0,
        "critical_failure": False,
        "events": [],
    },
    "audio_integrity": {
        "score": 0.88,
        "critical_failure": False,
        "events": [
            {"type": "abrupt_cutoff", "turn": 3, "critical": False},
        ],
    },
}

How Is It Calculated?

VoiceReliabilityMetric runs the responsiveness and audio-integrity checks and averages them evenly:

\text{Voice Reliability} = 0.5 \times \text{Responsiveness} + 0.5 \times \text{Audio Integrity}

A critical failure on either side forces the score to 0, regardless of how well the other half did. Missing agent audio, an undecodable clip, a badly looping buffer, a reply that never came, or a call that ended in a hangup, error, or idle timeout are all catastrophic — averaging them against a clean second half would hide exactly the failures this metric exists to surface.

The two halves are the same detectors used by AgentResponsivenessMetric and AudioIntegrityMetric, so their pages document what each event means and what it costs.

FAQs

Why is my score exactly 0 instead of somewhere in between?

A critical failure was detected. Check critical_failure on each half of the breakdown to see which side it came from, then read that half's events list for the specific failure.

Should I run this as well as the two metrics it combines?

Usually not both. Gate on this one and read its nested breakdown when it fails — the events it reports are identical to what the individual metrics would have given you. Run them separately when you want to threshold the two halves differently.

Does this metric call an LLM or an external speech model?

No. Both halves are deterministic detectors that run locally, so it has no model or token cost.

Can this metric be skipped for want of usable audio?

No. It always returns a score — a conversation with no assistant turns at all is a critical audio-integrity failure and scores 0, rather than being skipped as unevaluable. Only VoiceNaturalnessMetric, SpeechIntelligibilityMetric, VoiceConsistencyMetric, and TurnTakingNaturalnessMetric skip a test case, and only when there is nothing in it to measure.

Usage

How Is It Calculated?

FAQs

On this page