Audio Integrity

The audio integrity metric checks whether your voice agent's audio arrived intact. It looks for missing, undecodable, looping, dropping, clipped, or abruptly cut audio across the call and returns a score between 0 and 1, where 1 means no defects were found.

Usage

Voice metrics evaluate calls produced by the ConversationSimulator in voice mode. Start with one ConversationalGolden, simulate one call, then hand what it returns to evaluate():

from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import AudioIntegrityMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig


golden = ConversationalGolden(
    scenario="The user asks the agent to look up an order by its number.",
    expected_outcome="The agent reads back the order status.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(
        connector=ElevenLabsConnector(agent_id="your-agent-id"),
    )
)
test_cases = simulator.simulate([golden])

metric = AudioIntegrityMetric(threshold=0.8)
evaluate(test_cases=test_cases, metrics=[metric])

There are FIVE optional parameters when creating an AudioIntegrityMetric:

• [Optional] threshold: a number representing the minimum passing score. Set it to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] include_reason: a boolean which, when set to True, includes a summary of the evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which, when set to True, requires a perfect score. Any score below 1 becomes 0, and the threshold is set to 1. Defaulted to False.
• [Optional] verbose_mode: a boolean which, when set to True, includes the score breakdown in verbose metric logs. Defaulted to False.
• [Optional] flaky: a boolean which, when set to True, marks the metric as flaky. Defaulted to False.

Every voice metric records what it measured on score_breakdown. Read it after a standalone measure() call, or turn on verbose_mode to have it printed during an evaluate() run:

metric.measure(test_cases[0])

print(metric.score)  # e.g. 0.72
print(metric.score_breakdown)

Unlike VoiceNaturalnessMetric and SpeechIntelligibilityMetric, which score every turn, AudioIntegrityMetric reports a list of the defects it found:

{
    "critical_failure": False,
    "events": [
        {"type": "abrupt_cutoff", "turn": 3, "critical": False},
        {
            "type": "audio_dropout",
            "turn": 3,
            "count": 2,
            "severity": 0.16,
            "critical": False,
        },
    ],
}

turn is the turn's zero-based position in the conversation. An empty events list means nothing was detected.

How Is It Calculated?

AudioIntegrityMetric inspects every assistant turn and records an event for each defect it finds. A clean call scores 1; otherwise the score is 1 minus the summed severity of every defect, clamped at 0 — unless a critical event occurred, in which case the score is 0 outright:

Critical events zero the score because they cannot be averaged away by turns that happened to be fine:

Non-critical events subtract a bounded severity:

FAQs