Turn-Taking Naturalness
The turn-taking naturalness metric measures the rhythm of the call — how long each speaker waited before answering, and how much they talked over each other. It reconstructs the call timeline from the audio on every turn and returns a score between 0 and 1, where a higher score means smoother hand-offs.
Usage
Voice metrics evaluate calls produced by the ConversationSimulator in voice mode. Start with one ConversationalGolden, simulate one call, then hand what it returns to evaluate():
from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import TurnTakingNaturalnessMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig
golden = ConversationalGolden(
scenario="The user asks three short follow-up questions about a delivery.",
expected_outcome="The agent answers each question in turn.",
)
simulator = ConversationSimulator(
voice_config=VoiceConfig(
connector=ElevenLabsConnector(agent_id="your-agent-id"),
)
)
test_cases = simulator.simulate([golden])
metric = TurnTakingNaturalnessMetric(threshold=0.6)
evaluate(test_cases=test_cases, metrics=[metric])There are FIVE optional parameters when creating a TurnTakingNaturalnessMetric:
- [Optional]
threshold: a number representing the minimum passing score. Set it toNoneto run the metric in score-only mode. Defaulted to0.5. - [Optional]
include_reason: a boolean which, when set toTrue, includes a summary of the evaluation score. Defaulted toTrue. - [Optional]
strict_mode: a boolean which, when set toTrue, requires a perfect score. Any score below 1 becomes 0, and the threshold is set to 1. Defaulted toFalse. - [Optional]
verbose_mode: a boolean which, when set toTrue, includes the score breakdown in verbose metric logs. Defaulted toFalse. - [Optional]
flaky: a boolean which, when set toTrue, marks the metric as flaky. Defaulted toFalse.
Every voice metric records what it measured on score_breakdown. Read it after a standalone measure() call, or turn on verbose_mode to have it printed during an evaluate() run:
metric.measure(test_cases[0])
print(metric.score) # e.g. 0.78
print(metric.score_breakdown)For TurnTakingNaturalnessMetric the breakdown is one entry per speaker change:
{
"transitions": [
{
"from_turn": 0,
"to_turn": 1,
"kind": "gap",
"gap_seconds": 0.82,
"score": 0.72,
},
{
"from_turn": 1,
"to_turn": 2,
"kind": "user_barge_in",
"gap_seconds": -0.35,
"score": 0.84,
},
],
"response_delays": {
"count": 4,
"mean_seconds": 0.61,
"spread_seconds": 0.38,
"score": 0.83,
},
}A negative gap_seconds is overlap: the next speaker started before the previous one finished. response_delays summarises the caller-to-agent gaps only — how long the agent waited before replying each time, and how much that wait varied.
How Is It Calculated?
TurnTakingNaturalnessMetric places every clip on a call-relative timeline using Audio.start_time and Audio.duration, then scores each transition between consecutive turns with different roles. Same-speaker transitions are ignored, so an agent's two-part reply is not scored as a hand-off.
Each transition decays exponentially from a perfect 1 at zero delay, and the conversation score is their mean:
where T is the number of speaker changes, Δ is the gap or overlap in seconds, and the time constant τ is chosen by what kind of transition it is:
kind | Transition | τ |
|---|---|---|
gap | Silence between the two turns. | 2.5s |
user_barge_in | The caller started while the agent was talking. | 2.0s |
agent_overlap | The agent talked over the caller. | 0.6s |
A smaller τ decays faster. Agent-on-user overlap is the harshest because it is the most disruptive: half a second of it scores 0.43, where half a second of caller barge-in still scores 0.78.
There is no latency cutoff — the curve is continuous, so a 1.7s silence scores 0.5 and a 4s silence scores 0.2. Judge the score against your own baseline rather than expecting a natural call to reach 1.
Once the call has at least three caller-to-agent gaps, the metric also scores how consistently the agent waited before replying. An agent that jumps in after 2s once and waits 9s another time is applying its silence timeout unevenly, which the per-transition mean alone would hide. The spread σ of those gaps (their population standard deviation, in seconds) becomes a second term:
A perfectly even agent keeps the full score; a 1s spread costs about 0.08, a 3s spread about 0.16. With fewer than three caller-to-agent gaps the first term is the whole score and response_delays.score is None.
The metric returns no score and marks the evaluation as skipped when any audio turn is missing start_time, when fewer than two clips are placed on the timeline, or when no two consecutive turns change speaker.
FAQs
Why was my test case skipped?
start_time. Every clip needs a call-relative placement; without one, silence and overlap would have to be guessed from clip order, and the metric refuses to invent timing evidence. Test cases assembled by hand from recordings need start_time set explicitly.Is this the same as latency_ms?
latency_ms is one number per assistant turn — how long the agent took to start speaking. This metric scores every hand-off in both directions, including the caller's, and accounts for overlap that a latency figure cannot express.Do interruptions hurt the score?
Why does one slow reply lower the score more than its own transition suggests?
response_delays.score records that. The spread term is weighted at 20%, so it adjusts the score rather than dominating it.