Turn Contextual Recall

Q: When should I use TurnContextualRecallMetric instead of the single-turn version?

When retrieval feeds a multi-turn ConversationalTestCase and you want to confirm each turn retrieved everything needed for the . The single-turn ContextualRecallMetric checks one query/context pair only.

Q: Why does recall require expected outcome when relevancy doesn't?

Recall is measured against a reference: it breaks the into statements and checks how many map to a node in the . Without it there's nothing to test completeness against, which is why relevancy (referenceless) doesn't need it.

Q: My bot missed a detail the user needed — does a low recall confirm it's a retriever problem?

Largely, yes. Low recall means the never contained the info the needed, so the generator never had it. Fix the retriever (chunking, embeddings, top-k) before blaming the LLM.

LLM-as-a-judge

Multi-turn

RAG

Chatbot

Multimodal

The turn contextual recall metric is a conversational metric that evaluates whether the retrieval context contains sufficient information to support the expected outcome throughout a conversation.

Required Arguments

To use the TurnContextualRecallMetric, you'll have to provide the following arguments when creating a ConversationalTestCase:

turns
expected_outcome

You must provide the role, content, and retrieval_context for evaluation to happen. Read the How Is It Calculated section below to learn more.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The TurnContextualRecallMetric() can be used for end-to-end multi-turn evaluation:

from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import TurnContextualRecallMetric
from deepeval import evaluate

content = "We offer a 30-day full refund at no extra cost."
retrieval_context = [
    "All customers are eligible for a 30 day full refund at no extra cost."
]

convo_test_case = ConversationalTestCase(
    turns=[
        Turn(role="user", content="What if these shoes don't fit?"),
        Turn(role="assistant", content=content, retrieval_context=retrieval_context)
    ],
    expected_outcome="The chatbot must explain the store policies like refunds, discounts, ..etc.",
)

metric = TurnContextualRecallMetric(threshold=0.5)

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[metric])

There are TEN optional parameters when creating a TurnContextualRecallMetric:

[Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
[Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] window_size: an integer which defines the size of the sliding window of turns used during evaluation. Defaulted to 10.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
[Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
[Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

As a standalone

You can also run the TurnContextualRecallMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)

How Is It Calculated?

You can change how the TurnContextualRecallMetric is calculated by setting the eval mode.

LLM-as-a-judge

The TurnContextualRecallMetric score is calculated according to the following equation:

\text{Turn Contextual Recall} = \frac{\sum \text{Turn Contextual Recall Scores}}{\text{Total Number of Assistant Turns}}

The TurnContextualRecallMetric first constructs a sliding windows of turns. For each window, it:

Breaks down the expected outcome into individual sentences or statements
Evaluates each sentence to determine if it can be attributed to any node in the retrieval context
Calculates the interaction score as the ratio of attributable sentences to total sentences

\text{Contextual Recall} = \frac{\text{Number of Attributable Statements}}{\text{Total Number of Statements}}

The final score is the average of all recall scores across the conversation. This measures whether your retrieval system is providing sufficient information to generate the expected responses.

Hybrid

Under the hybrid eval mode, the expected_outcome is split into sentences in code for each window, and Jev, a System One model, answers one yes/no question per sentence: can it be attributed to that window's retrieval_context? P(yes) >= 0.5 counts as attributable. The equation and the LLM-written reasons are unchanged. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the whole conversation (with each assistant turn's retrieval_context) and the expected_outcome and asked three questions:

Question	Type	Weight
Every sentence in `expected_outcome` can be attributed to the facts in the `retrieval_context` of the assistant turns in `turns`.	Noul	2
The `retrieval_context` of the assistant turns in `turns` contains the key facts needed to reach `expected_outcome`.	Noul	1
How much of `expected_outcome` can be attributed to the facts in the `retrieval_context` of the assistant turns in `turns`? (None of it → All of it)	Score	1

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs

When should I use TurnContextualRecallMetric instead of the single-turn version?

When retrieval feeds a multi-turn ConversationalTestCase and you want to confirm each turn retrieved everything needed for the expected_outcome. The single-turn ContextualRecallMetric checks one query/context pair only.

Why does recall require expected outcome when relevancy doesn't?

Recall is measured against a reference: it breaks the expected_outcome into statements and checks how many map to a node in the retrieval_context. Without it there's nothing to test completeness against, which is why relevancy (referenceless) doesn't need it.

How is recall different from relevancy and precision?

Recall measures completeness, relevancy measures signal vs noise, and precision measures ranking.

My bot missed a detail the user needed — does a low recall confirm it's a retriever problem?

Largely, yes. Low recall means the retrieval_context never contained the info the expected_outcome needed, so the generator never had it. Fix the retriever (chunking, embeddings, top-k) before blaming the LLM.

On this page