Turn-Taking Naturalness

The turn-taking naturalness metric measures the rhythm of the call — how long each speaker waited before answering, and how much they talked over each other. It reconstructs the call timeline from the audio on every turn and returns a score between 0 and 1, where a higher score means smoother hand-offs.

Usage

Voice metrics evaluate calls produced by the ConversationSimulator in voice mode. Start with one ConversationalGolden, simulate one call, then hand what it returns to evaluate():

from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import TurnTakingNaturalnessMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig


golden = ConversationalGolden(
    scenario="The user asks three short follow-up questions about a delivery.",
    expected_outcome="The agent answers each question in turn.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(
        connector=ElevenLabsConnector(agent_id="your-agent-id"),
    )
)
test_cases = simulator.simulate([golden])

metric = TurnTakingNaturalnessMetric(threshold=0.6)
evaluate(test_cases=test_cases, metrics=[metric])

There are FIVE optional parameters when creating a TurnTakingNaturalnessMetric:

• [Optional] threshold: a number representing the minimum passing score. Set it to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] include_reason: a boolean which, when set to True, includes a summary of the evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which, when set to True, requires a perfect score. Any score below 1 becomes 0, and the threshold is set to 1. Defaulted to False.
• [Optional] verbose_mode: a boolean which, when set to True, includes the score breakdown in verbose metric logs. Defaulted to False.
• [Optional] flaky: a boolean which, when set to True, marks the metric as flaky. Defaulted to False.

Every voice metric records what it measured on score_breakdown. Read it after a standalone measure() call, or turn on verbose_mode to have it printed during an evaluate() run:

metric.measure(test_cases[0])

print(metric.score)  # e.g. 0.78
print(metric.score_breakdown)

For TurnTakingNaturalnessMetric the breakdown is one entry per speaker change:

{
    "transitions": [
        {
            "from_turn": 0,
            "to_turn": 1,
            "kind": "gap",
            "gap_seconds": 0.82,
            "score": 0.72,
        },
        {
            "from_turn": 1,
            "to_turn": 2,
            "kind": "user_barge_in",
            "gap_seconds": -0.35,
            "score": 0.84,
        },
    ],
}

A negative gap_seconds is overlap: the next speaker started before the previous one finished.

How Is It Calculated?

TurnTakingNaturalnessMetric places every clip on a call-relative timeline using Audio.start_time and Audio.duration, then scores each transition between consecutive turns with different roles. Same-speaker transitions are ignored, so an agent's two-part reply is not scored as a hand-off.

Each transition decays exponentially from a perfect 1 at zero delay, and the conversation score is their mean:

where T is the number of speaker changes, Δ is the gap or overlap in seconds, and the time constant τ is chosen by what kind of transition it is:

A smaller τ decays faster. Agent-on-user overlap is the harshest because it is the most disruptive: half a second of it scores 0.43, where half a second of caller barge-in still scores 0.78.

There is no latency cutoff — the curve is continuous, so a 1.7s silence scores 0.5 and a 4s silence scores 0.2. Judge the score against your own baseline rather than expecting a natural call to reach 1.

The metric returns no score and marks the evaluation as skipped when any audio turn is missing start_time, when fewer than two clips are placed on the timeline, or when no two consecutive turns change speaker.

FAQs