Agent Responsiveness

The agent responsiveness metric checks whether your voice agent answered when it was spoken to. It walks the conversation in order looking for turns the caller had to repeat, replies that never came, and calls that ended badly, returning a score between 0 and 1.

Usage

Voice metrics evaluate calls produced by the ConversationSimulator in voice mode. Start with one ConversationalGolden, simulate one call, then hand what it returns to evaluate():

from deepeval import evaluate
from deepeval.dataset import ConversationalGolden
from deepeval.metrics import AgentResponsivenessMetric
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig


golden = ConversationalGolden(
    scenario="The user asks a question that requires the agent to look up an account.",
    expected_outcome="The agent answers without the user having to repeat themselves.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(
        connector=ElevenLabsConnector(agent_id="your-agent-id"),
    )
)
test_cases = simulator.simulate([golden])

metric = AgentResponsivenessMetric(threshold=0.8)
evaluate(test_cases=test_cases, metrics=[metric])

There are FIVE optional parameters when creating an AgentResponsivenessMetric:

• [Optional] threshold: a number representing the minimum passing score. Set it to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] include_reason: a boolean which, when set to True, includes a summary of the evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which, when set to True, requires a perfect score. Any score below 1 becomes 0, and the threshold is set to 1. Defaulted to False.
• [Optional] verbose_mode: a boolean which, when set to True, includes the score breakdown in verbose metric logs. Defaulted to False.
• [Optional] flaky: a boolean which, when set to True, marks the metric as flaky. Defaulted to False.

Every voice metric records what it measured on score_breakdown. Read it after a standalone measure() call, or turn on verbose_mode to have it printed during an evaluate() run:

metric.measure(test_cases[0])

print(metric.score)  # e.g. 0.75
print(metric.score_breakdown)

For AgentResponsivenessMetric the breakdown is a list of the events that were detected:

{
    "critical_failure": False,
    "events": [
        {"type": "user_reprompted", "turn": 2, "critical": False},
    ],
}

turn is the zero-based position of the offending turn in the conversation. An empty events list means the agent answered every time it was addressed.

How Is It Calculated?

AgentResponsivenessMetric looks at each user turn that owed a reply, and checks what came next. A user turn that reads as a sign-off — one ending in "bye", "goodbye", "thanks", "thank you", or "that's all" — owes nothing, so a call that ends on a pleasantry is not marked as a failure.

Each remaining user turn produces one of these outcomes:

Any critical event forces the score to 0. Otherwise each reprompt costs 0.25, so four reprompts in a single call reach 0:

FAQs