Voice Naturalness

The voice naturalness metric measures how naturally your voice agent speaks across a conversation. It analyzes the audio on each assistant turn and returns a score between 0 and 1, where a higher score indicates more natural-sounding speech.

Usage

Voice metrics evaluate calls produced by the ConversationSimulator in voice mode. Start with one ConversationalGolden, simulate one call, then evaluate the returned test case:

from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import VoiceNaturalnessMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig


golden = ConversationalGolden(
    scenario="The user wants to change a restaurant reservation.",
    expected_outcome="The agent confirms the new reservation date.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(
        connector=ElevenLabsConnector(agent_id="your-agent-id"),
    )
)
test_case = simulator.simulate([golden])[0]

metric = VoiceNaturalnessMetric(threshold=0.7)
evaluate(test_cases=[test_case], metrics=[metric])

The simulator records the audio and transcript on each turn. You do not need to manually construct a ConversationalTestCase or Audio object.

There are FIVE optional parameters when creating a VoiceNaturalnessMetric:

• [Optional] threshold: a number representing the minimum passing score. Set it to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] include_reason: a boolean which, when set to True, includes a summary of the evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which, when set to True, requires a perfect score. Any score below 1 becomes 0, and the threshold is set to 1. Defaulted to False.
• [Optional] verbose_mode: a boolean which, when set to True, includes the score breakdown in verbose metric logs. Defaulted to False.
• [Optional] flaky: a boolean which, when set to True, marks the metric as flaky. Defaulted to False.

The metric's per-turn diagnostics include:

{
    "eligible_turns": 1,
    "turns": [
        {
            "turn": 1,
            "score": 0.91,
            "speaking_rate_wpm": 154.3,
            "silence_fraction": 0.08,
            "pitch_variation_hz": 22.7,
            "clipping_fraction": 0.0,
            "dropout_events": 0,
            "loop_events": 0,
        }
    ],
}

How Is It Calculated?

VoiceNaturalnessMetric evaluates assistant turns only. For each decodable assistant audio clip, it:

1. Decodes the clip to mono PCM samples.
2. Measures clipping, short dropouts, repeated audio windows, silence, estimated signal-to-noise ratio, speaking rate, and pitch variation.
3. Starts the turn at a score of 1 and applies bounded penalties for unnatural acoustic behavior.
4. Clamps the turn score between 0 and 1.
5. Averages all eligible assistant-turn scores.

The largest clipping penalty is 0.35. Dropouts can subtract up to 0.25, and repeated audio can subtract up to 0.2. Smaller penalties apply to:

• silence occupying more than 45% of the clip
• an estimated signal-to-noise ratio below 15 dB
• a speaking rate below 80 or above 240 words per minute
• pitch variation below 4 Hz or above 90 Hz

If an assistant turn has no audio, the metric skips that turn. If no assistant turn contains decodable audio, the metric returns no score and marks the evaluation as skipped.

FAQs