📍 Join the 1st DeepEval meetup in SF, this Friday 5pm PT. RSVP on Luma →
💥 Introducing JevEval: Jev-as-a-Judge for LLM evaluation. Read the post →
Voice Agents

Turn-Taking Naturalness

Beta
Multi-turn
Referenceless
Chatbot

The turn-taking naturalness metric measures the rhythm of the call — how long each speaker waited before answering, and how much they talked over each other. It reconstructs the call timeline from the audio on every turn and returns a score between 0 and 1, where a higher score means smoother hand-offs.

Usage

Voice metrics evaluate calls produced by the ConversationSimulator in voice mode. Start with one ConversationalGolden, simulate one call, then hand what it returns to evaluate():

from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import TurnTakingNaturalnessMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig


golden = ConversationalGolden(
    scenario="The user asks three short follow-up questions about a delivery.",
    expected_outcome="The agent answers each question in turn.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(
        connector=ElevenLabsConnector(agent_id="your-agent-id"),
    )
)
test_cases = simulator.simulate([golden])

metric = TurnTakingNaturalnessMetric(threshold=0.6)
evaluate(test_cases=test_cases, metrics=[metric])

There are FIVE optional parameters when creating a TurnTakingNaturalnessMetric:

  • [Optional] threshold: a number representing the minimum passing score. Set it to None to run the metric in score-only mode. Defaulted to 0.5.
  • [Optional] include_reason: a boolean which, when set to True, includes a summary of the evaluation score. Defaulted to True.
  • [Optional] strict_mode: a boolean which, when set to True, requires a perfect score. Any score below 1 becomes 0, and the threshold is set to 1. Defaulted to False.
  • [Optional] verbose_mode: a boolean which, when set to True, includes the score breakdown in verbose metric logs. Defaulted to False.
  • [Optional] flaky: a boolean which, when set to True, marks the metric as flaky. Defaulted to False.

Every voice metric records what it measured on score_breakdown. Read it after a standalone measure() call, or turn on verbose_mode to have it printed during an evaluate() run:

metric.measure(test_cases[0])

print(metric.score)  # e.g. 0.78
print(metric.score_breakdown)

For TurnTakingNaturalnessMetric the breakdown is one entry per speaker change:

{
    "transitions": [
        {
            "from_turn": 0,
            "to_turn": 1,
            "kind": "gap",
            "gap_seconds": 0.82,
            "score": 0.72,
        },
        {
            "from_turn": 1,
            "to_turn": 2,
            "kind": "user_barge_in",
            "gap_seconds": -0.35,
            "score": 0.84,
        },
    ],
    "response_delays": {
        "count": 4,
        "mean_seconds": 0.61,
        "spread_seconds": 0.38,
        "score": 0.83,
    },
}

A negative gap_seconds is overlap: the next speaker started before the previous one finished. response_delays summarises the caller-to-agent gaps only — how long the agent waited before replying each time, and how much that wait varied.

How Is It Calculated?

TurnTakingNaturalnessMetric places every clip on a call-relative timeline using Audio.start_time and Audio.duration, then scores each transition between consecutive turns with different roles. Same-speaker transitions are ignored, so an agent's two-part reply is not scored as a hand-off.

Each transition decays exponentially from a perfect 1 at zero delay, and the conversation score is their mean:

Turn-Taking Naturalness=1T∑t=1Te−∣Δt∣/τt\text{Turn-Taking Naturalness} = \frac{1}{T} \sum_{t=1}^{T} e^{-|\Delta_t| / \tau_t}

where T is the number of speaker changes, Δ is the gap or overlap in seconds, and the time constant τ is chosen by what kind of transition it is:

kindTransitionτ
gapSilence between the two turns.2.5s
user_barge_inThe caller started while the agent was talking.2.0s
agent_overlapThe agent talked over the caller.0.6s

A smaller τ decays faster. Agent-on-user overlap is the harshest because it is the most disruptive: half a second of it scores 0.43, where half a second of caller barge-in still scores 0.78.

There is no latency cutoff — the curve is continuous, so a 1.7s silence scores 0.5 and a 4s silence scores 0.2. Judge the score against your own baseline rather than expecting a natural call to reach 1.

Once the call has at least three caller-to-agent gaps, the metric also scores how consistently the agent waited before replying. An agent that jumps in after 2s once and waits 9s another time is applying its silence timeout unevenly, which the per-transition mean alone would hide. The spread σ of those gaps (their population standard deviation, in seconds) becomes a second term:

Turn-Taking Naturalness=0.8⋅1T∑t=1Te−∣Δt∣/τt+0.2⋅e−σ/2\text{Turn-Taking Naturalness} = 0.8 \cdot \frac{1}{T} \sum_{t=1}^{T} e^{-|\Delta_t| / \tau_t} + 0.2 \cdot e^{-\sigma / 2}

A perfectly even agent keeps the full score; a 1s spread costs about 0.08, a 3s spread about 0.16. With fewer than three caller-to-agent gaps the first term is the whole score and response_delays.score is None.

The metric returns no score and marks the evaluation as skipped when any audio turn is missing start_time, when fewer than two clips are placed on the timeline, or when no two consecutive turns change speaker.

FAQs

Why was my test case skipped?
Almost always a missing start_time. Every clip needs a call-relative placement; without one, silence and overlap would have to be guessed from clip order, and the metric refuses to invent timing evidence. Test cases assembled by hand from recordings need start_time set explicitly.
Is this the same as latency_ms?
Related but not the same. latency_ms is one number per assistant turn — how long the agent took to start speaking. This metric scores every hand-off in both directions, including the caller's, and accounts for overlap that a latency figure cannot express.
Do interruptions hurt the score?
Mildly, by design. A caller barging in is normal phone behavior, so it decays slowly; the agent failing to yield decays more than three times as fast. Enabling interruptions is what makes those transitions appear at all.
Why does one slow reply lower the score more than its own transition suggests?
Because it also widens the spread of the agent's response delays. Four replies at 0.4s and one at 3s score reasonably well transition by transition, but the agent plainly did not apply the same timeout each time, and response_delays.score records that. The spread term is weighted at 20%, so it adjusts the score rather than dominating it.
Does this metric call an LLM or an external speech model?
No. Scoring is deterministic and reads clip timings, so it has no model or token cost.

On this page