Voice Naturalness

Q: Do I need a transcript?

Yes. The assistant turn's content is used with the audio duration and detected silence to estimate speaking rate.

Q: Does the metric evaluate user audio?

No. Only turns whose role is "assistant" contribute to the score.

Multi-turn

Referenceless

Chatbot

The voice naturalness metric measures how naturally your voice agent speaks across a conversation. It analyzes the audio on each assistant turn and returns a score between 0 and 1, where a higher score indicates more natural-sounding speech.

Usage

Voice metrics evaluate calls produced by the ConversationSimulator in voice mode. Start with one ConversationalGolden, simulate one call, then hand what it returns to evaluate():

from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import VoiceNaturalnessMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig


golden = ConversationalGolden(
    scenario="The user wants to change a restaurant reservation.",
    expected_outcome="The agent confirms the new reservation date.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(
        connector=ElevenLabsConnector(agent_id="your-agent-id"),
    )
)
test_cases = simulator.simulate([golden])

metric = VoiceNaturalnessMetric(threshold=0.7)
evaluate(test_cases=test_cases, metrics=[metric])

There are FIVE optional parameters when creating a VoiceNaturalnessMetric:

[Optional] threshold: a number representing the minimum passing score. Set it to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] include_reason: a boolean which, when set to True, includes a summary of the evaluation score. Defaulted to True.
[Optional] strict_mode: a boolean which, when set to True, requires a perfect score. Any score below 1 becomes 0, and the threshold is set to 1. Defaulted to False.
[Optional] verbose_mode: a boolean which, when set to True, includes the score breakdown in verbose metric logs. Defaulted to False.
[Optional] flaky: a boolean which, when set to True, marks the metric as flaky. Defaulted to False.

Every voice metric records what it measured on score_breakdown. Read it after a standalone measure() call, or turn on verbose_mode to have it printed during an evaluate() run:

metric.measure(test_cases[0])

print(metric.score)  # e.g. 0.91
print(metric.score_breakdown)

For VoiceNaturalnessMetric the breakdown is one entry per assistant turn:

{
    "eligible_turns": 1,
    "turns": [
        {
            "turn": 1,
            "score": 0.91,
            "speaking_rate_wpm": 154.3,
            "silence_fraction": 0.08,
            "pitch_variation_hz": 22.7,
            "clipping_fraction": 0.0,
            "dropout_events": 0,
            "loop_events": 0,
        }
    ],
}

How Is It Calculated?

VoiceNaturalnessMetric evaluates assistant turns only. Each decodable assistant clip is decoded to mono PCM samples, starts at a perfect 1, and loses a bounded penalty for every unnatural acoustic behavior found in it. The conversation score is the mean of those turn scores:

\text{Voice Naturalness} = \frac{1}{N} \sum_{i=1}^{N} \max \left( 0,\ 1 - \sum_{p} \text{penalty}_p(i) \right)

where N is the number of eligible assistant turns and p ranges over these penalties:

Penalty	Applies when	Maximum
Clipping	Samples hit the ceiling of the format.	0.35
Dropouts	Short silences interrupt continuous speech.	0.25
Repeated audio	The same 0.25s window recurs — a stuck buffer.	0.2
Excess silence	Silence occupies more than 45% of the clip.	0.2
Low SNR	Estimated signal-to-noise ratio falls below 15 dB.	0.2
Speaking rate	Speech runs below 80 or above 240 words per minute.	0.2
Pitch variation	Pitch varies by less than 4 Hz or more than 90 Hz.	0.1

Every penalty is graded by how far past its trigger the measurement went, except pitch variation, which is a flat 0.1.

If an assistant turn has no audio, the metric skips that turn. If no assistant turn contains decodable audio, the metric returns no score and marks the evaluation as skipped.

FAQs

Does this metric call an LLM or an external speech model?

No. Scoring is deterministic and runs locally over the audio samples, so it has no model or token cost.

Do I need a transcript?

Yes. The assistant turn's content is used with the audio duration and detected silence to estimate speaking rate.

Does the metric evaluate user audio?

No. Only turns whose role is "assistant" contribute to the score.

Is this a replacement for human listening tests?

No. It is a fast regression signal for measurable acoustic defects. Use representative human ratings when you need a perceptual quality benchmark or must validate a new voice and speaking style.

Usage

How Is It Calculated?

FAQs

On this page