Contextual Recall

Q: Does contextual recall tell me if I'm missing relevant chunks?

Yes — exactly what it measures. It breaks into statements and checks how many trace back to a node in . Any statement that can't signals information your retriever failed to surface.

Q: My contextual recall is low — which retriever knobs should I tune?

Needed information never made it into the . Usual levers: increase top-K, adjust chunk size so facts aren't split, and swap in a stronger embedding model so relevant passages rank within your cutoff.

LLM-as-a-judge

Jev-as-a-judge

Single-turn

Reference-based

RAG

Multimodal

The contextual recall metric uses LLM-as-a-judge to measure the quality of your RAG pipeline's retriever by evaluating the extent of which the retrieval_context aligns with the expected_output. deepeval's contextual recall metric is a self-explaining LLM-Eval, meaning it outputs a reason for its metric score.

Required Arguments

To use the ContextualRecallMetric, you'll have to provide the following arguments when creating an LLMTestCase:

input
actual_output
expected_output
retrieval_context

Read the How Is It Calculated section below to learn how test case parameters are used for metric calculation.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The ContextualRecallMetric() can be used for end-to-end evaluation of text-based and multimodal test cases:

from deepeval.metrics import ContextualRecallMetric
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the expected output from your RAG generator
expected_output = "You are eligible for a 30 day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = ContextualRecallMetric(
    threshold=0.7,
    model="gpt-4",
    include_reason=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

from deepeval.test_case import LLMTestCase, MLLMImage
from deepeval.metrics import ContextualRecallMetric
from deepeval import evaluate

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = [
    f"The Eiffel Tower {MLLMImage(...)} is a wrought-iron lattice tower built in the late 19th century.",
    f"...",
]

metric = ContextualRecallMetric(
    threshold=0.7,
    model="gpt-4.1",
    include_reason=True
)
test_case = LLMTestCase(
    input=f"Tell me about this landmark in France: {MLLMImage(...)}",
    actual_output=f"This appears to be Eiffel Tower, which is a famous landmark in France"
    expected_output=f"The Eiffel Tower is located in Paris, France. {MLLMImage(...)}",
    retrieval_context=retrieval_context
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

There are TEN optional parameters when creating a ContextualRecallMetric:

[Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
[Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] evaluation_template: a class of type ContextualRecallTemplate, which allows you to override the default prompts used to compute the ContextualRecallMetric score. Defaulted to deepeval's ContextualRecallTemplate.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
[Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
[Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

Within components

You can also run the ContextualRecallMetric within nested components for component-level evaluation.

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.tracing import observe, update_current_span
...

@observe(metrics=[metric])
def inner_component():
    # Set test case at runtime
    test_case = LLMTestCase(input="...", actual_output="...")
    update_current_span(test_case=test_case)
    return

@observe
def llm_app(input: str):
    # Component can be anything from an LLM call, retrieval, agent, tool use, etc.
    inner_component()
    return

dataset = EvaluationDataset(goldens=[Golden(input="Hi!")])
for golden in dataset.evals_iterator():
    llm_app(golden.input)

As a standalone

You can also run the ContextualRecallMetric on a single test case as a standalone, one-off execution.

...

metric.measure(test_case)
print(metric.score, metric.reason)

How Is It Calculated?

You can change how the ContextualRecallMetric is calculated by setting the eval mode.

LLM-as-a-judge

The ContextualRecallMetric score is calculated according to the following equation:

\text{Contextual Recall} = \frac{\text{Number of Attributable Statements}}{\text{Total Number of Statements}}

The ContextualRecallMetric first uses an LLM to extract all statements made in the expected_output, before using the same LLM to classify whether each statement can be attributed to nodes in the retrieval_context.

A higher contextual recall score represents a greater ability of the retrieval system to capture all relevant information from the total available relevant set within your knowledge base.

Hybrid

Under the hybrid eval mode, the expected_output is split into sentences in code, and Jev, a System One model, answers one yes/no question per sentence: can it be attributed to the retrieval_context? P(yes) >= 0.5 counts as attributable. The equation and the LLM-written reason are unchanged. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the raw expected_output and retrieval_context and asked three questions:

Question	Type	Weight
Every sentence in `expected_output` can be attributed to the facts in `retrieval_context`.	Noul	2
`retrieval_context` contains the key facts needed to write `expected_output`.	Noul	1
How much of `expected_output` can be attributed to the facts in `retrieval_context`? (None of it → All of it)	Score	1

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

Customize Your Template

Since deepeval's ContextualRecallMetric is evaluated by LLM-as-a-judge, you can likely improve your metric accuracy by overriding deepeval's default prompt templates. This is especially helpful if:

You're using a custom evaluation LLM, especially for smaller models that have weaker instruction following capabilities.
You want to customize the examples used in the default ContextualRecallTemplate to better align with your expectations.

Here's a quick example of how you can override the relevancy classification step of the ContextualRecallMetric algorithm:

from deepeval.metrics.contextual_recall import ContextualRecallTemplate
from deepeval.metrics import ContextualRecallMetric

# Define custom template
class CustomTemplate(ContextualRecallTemplate):
    @staticmethod
    def generate_verdicts(expected_output: str, retrieval_context: List[str]):
        return f"""For EACH sentence in the given expected output below, determine whether the sentence can be attributed to the nodes of retrieval contexts.

Example JSON:
{{
    "verdicts": [
        {{
            "verdict": "yes",
            "reason": "..."
        }},
    ]
}}

Expected Output:
{expected_output}

Retrieval Context:
{retrieval_context}

JSON:
"""

# Inject custom template to metric
metric = ContextualRecallMetric(evaluation_template=CustomTemplate)
metric.measure(...)

FAQs

Does contextual recall tell me if I'm missing relevant chunks?

Yes — exactly what it measures. It breaks expected_output into statements and checks how many trace back to a node in retrieval_context. Any statement that can't signals information your retriever failed to surface.

My contextual recall is low — which retriever knobs should I tune?

Needed information never made it into the retrieval_context. Usual levers: increase top-K, adjust chunk size so facts aren't split, and swap in a stronger embedding model so relevant passages rank within your cutoff.

Why does it use expected output instead of actual output?

Recall measures whether the retriever fetched everything needed for the ideal answer, so it attributes expected_output (ground truth) to the retrieval_context. Using actual_output would let a bad answer unfairly drag down a good retriever.

How is Contextual Recall different from Contextual Precision?

Recall is completeness — did the retriever pull in all relevant information? Contextual Precision is ordering — are relevant chunks ranked above irrelevant ones? Want both high.

On this page