Turn Faithfulness

MLLM-as-a-judge

Multi-turn

Chatbot

Multimodal

The turn faithfulness metric is a conversational metric that determines whether your LLM chatbot generates factually accurate responses grounded in the retrieval context throughout a conversation.

Required Arguments

To use the TurnFaithfulnessMetric, you'll have to provide the following arguments when creating a ConversationalTestCase:

• turns

You must provide the role, content, and retrieval_context for evaluation to happen. Read the How Is It Calculated section below to learn more.

Usage

The TurnFaithfulnessMetric() can be used for end-to-end multi-turn evaluation:

from deepeval import evaluate
from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import TurnFaithfulnessMetric

convo_test_case = ConversationalTestCase(
    turns=[
        Turn(role="user", content="...", retrieval_context=["..."]),
        Turn(role="assistant", content="...", retrieval_context=["..."])
    ]
)
metric = TurnFaithfulnessMetric(threshold=0.5)

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[metric])

There are NINE optional parameters when creating a TurnFaithfulnessMetric:

• [Optional] threshold: a float representing the minimum passing threshold, defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to 'gpt-4.1'.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] truths_extraction_limit: an optional integer to limit the number of truths extracted from retrieval context per document. Defaulted to None.
• [Optional] penalize_ambiguous_claims: a boolean which when set to True, penalizes claims that cannot be verified as true or false. Defaulted to False.
• [Optional] window_size: an integer which defines the size of the sliding window of turns used during evaluation. Defaulted to 10.

As a standalone

You can also run the TurnFaithfulnessMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)

caution

This is great for debugging or if you wish to build your own evaluation pipeline, but you will NOT get the benefits (testing reports, Confident AI platform) and all the optimizations (speed, caching, computation) the evaluate() function or deepeval test run offers.

How Is It Calculated?

The TurnFaithfulnessMetric score is calculated according to the following equation:

$$\text{Turn Faithfulness} = \frac{\sum \text{Turn Faithfulness Scores}}{\text{Total Number of Assistant Turns}}$$

The TurnFaithfulnessMetric first constructs a sliding windows of turns. For each window, it:

1. Extracts truths from the retrieval context provided in the turns
2. Generates claims from the assistant's responses in the interaction
3. Evaluates verdicts by checking if each claim contradicts the truths
4. Calculates the interaction score as the ratio of faithful claims to total claims

$$\text{Faithfulness} = \frac{\text{Number of Truthful Claims}}{\text{Total Number of Claims}}$$

The final score is the average of all interaction faithfulness scores across the conversation.