Transcription Accuracy

The transcription accuracy metric measures whether your voice agent's own speech-to-text heard the caller correctly. It compares what the caller said against the transcription the agent's connector reported for that speech, so a reply that looks wrong can be traced to a mis-heard word rather than to the LLM behind it.

Required Arguments

To use the TranscriptionAccuracyMetric, you'll have to provide the following arguments when creating a ConversationalTestCase:

• turns

You must provide the role and content for every turn, and provider_transcript on at least one "assistant" turn. Read the How Is It Calculated section below to learn more.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The TranscriptionAccuracyMetric() can be used for end-to-end multi-turn evaluation:

from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import TranscriptionAccuracyMetric
from deepeval import evaluate

convo_test_case = ConversationalTestCase(
    turns=[
        Turn(role="user", content="Siobhan Kavanagh here, I can start in two weeks."),
        Turn(
            role="assistant",
            content="Great to meet you Siobhan, two weeks works.",
            provider_transcript="Siobhan Cavanaugh here, I can start in two weeks.",
        ),
    ]
)
metric = TranscriptionAccuracyMetric(threshold=0.5)

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[metric])

In practice you won't write those turns by hand — the ConversationSimulator in voice mode fills them in for you, with content holding what the simulated caller was scripted to say and provider_transcript holding what the agent reported hearing:

from deepeval.dataset import ConversationalGolden
from deepeval.simulator import ConversationSimulator
from deepeval.voice import ElevenLabsConnector, VoiceConfig

golden = ConversationalGolden(
    scenario="A recruiter's voice agent screens a candidate for a backend role.",
    expected_outcome="The candidate states their name and their notice period.",
)

simulator = ConversationSimulator(
    voice_config=VoiceConfig(
        connector=ElevenLabsConnector(agent_id="your-agent-id"),
    )
)
test_cases = simulator.simulate([golden])

evaluate(test_cases=test_cases, metrics=[TranscriptionAccuracyMetric()])

There are NINE optional parameters when creating a TranscriptionAccuracyMetric:

• [Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
• [Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
• [Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

As a standalone

You can also run the TranscriptionAccuracyMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)

How Is It Calculated?

The metric first pairs the conversation into exchanges. Each "assistant" turn carrying a provider_transcript is paired with every "user" turn since the agent last spoke, joined together — a barge-in splits one piece of caller speech across two user turns, and the agent's transcription covers all of it. An assistant turn with no transcription is unmeasurable, and the caller speech before it is dropped with it.

You can change how the TranscriptionAccuracyMetric is calculated by setting the eval mode.

LLM-as-a-judge

The TranscriptionAccuracyMetric score is calculated according to the following equation:

An LLM judges each exchange, and deliberately does not count these as errors, because they are the same utterance written differently:

• Punctuation, capitalisation and sentence splitting — "Yes, could you" vs "Yes. Could you"
• Numbers as digits rather than words — "fifteen years" vs "15 years"
• Dropped filler words, and the same utterance emitted twice by the speech-to-text engine

It does count these, because they change what the agent was told:

• A name or other proper noun heard as a different word — "Kavanagh" vs "Cavanaugh"
• A number, date or duration whose value differs — "fifteen" vs "fifty", "two weeks" vs "two months"
• A negation gained or lost — "I can't start" vs "I can start"
• An empty transcription where the caller spoke, which means the agent heard nothing at all

Hybrid

Under the hybrid eval mode, the verdict for each exchange is answered by Jev, a System One model, instead: one yes/no question per exchange on whether the transcription captures what was spoken, with P(yes) >= 0.5 counted as faithful. The equation and the LLM-written reason are unchanged, and the reason still quotes the words that differed. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the paired exchanges and asked three questions:

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs