Turn Contextual Precision

Q: When should I use TurnContextualPrecisionMetric instead of the single-turn version?

When retrieval runs per turn in a ConversationalTestCase and you want to grade each turn's node ranking against the . The single-turn ContextualPrecisionMetric grades one retrieval only.

Q: How is precision different from recall and relevancy?

Precision is ranking — are relevant nodes ranked above irrelevant ones, weighted so earlier nodes matter more. Recall measures completeness and relevancy measures signal vs noise.

Q: Why does this metric need expected outcome?

It uses the to decide which nodes in the were useful before scoring their ranking. Without it there's no ground truth for labeling nodes relevant or irrelevant.

LLM-as-a-judge

Jev-as-a-judge

Multi-turn

RAG

Chatbot

Multimodal

The turn contextual precision metric is a conversational metric that evaluates whether relevant nodes in your retrieval context are ranked higher than irrelevant nodes throughout a conversation.

Required Arguments

To use the TurnContextualPrecisionMetric, you'll have to provide the following arguments when creating a ConversationalTestCase:

turns
expected_outcome

You must provide the role, content, and retrieval_context for evaluation to happen. Read the How Is It Calculated section below to learn more.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The TurnContextualPrecisionMetric() can be used for end-to-end multi-turn evaluation:

from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import TurnContextualPrecisionMetric
from deepeval import evaluate

content = "We offer a 30-day full refund at no extra cost."
retrieval_context = [
    "All customers are eligible for a 30 day full refund at no extra cost."
]

convo_test_case = ConversationalTestCase(
    turns=[
        Turn(role="user", content="What if these shoes don't fit?"),
        Turn(role="assistant", content=content, retrieval_context=retrieval_context)
    ],
    expected_outcome="The chatbot must explain the store policies like refunds, discounts, ..etc.",
)

metric = TurnContextualPrecisionMetric(threshold=0.5)

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[metric])

There are TEN optional parameters when creating a TurnContextualPrecisionMetric:

[Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
[Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] window_size: an integer which defines the size of the sliding window of turns used during evaluation. Defaulted to 10.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
[Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
[Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

As a standalone

You can also run the TurnContextualPrecisionMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)

How Is It Calculated?

You can change how the TurnContextualPrecisionMetric is calculated by setting the eval mode.

LLM-as-a-judge

The TurnContextualPrecisionMetric score is calculated according to the following equation:

\text{Turn Contextual Precision} = \frac{\sum \text{Turn Contextual Precision Scores}}{\text{Total Number of Assistant Turns}}

The TurnContextualPrecisionMetric first constructs a sliding windows of turns. For each window, it:

Evaluates each retrieval context node to determine if it was useful in arriving at the expected outcome
Calculates weighted precision where earlier relevant nodes contribute more to the score:

\text{Contextual Precision} = \frac{1}{\text{Number of Relevant Nodes}} \sum_{k=1}^{n} \left( \frac{\text{Number of Relevant Nodes Up to Position } k}{k} \times r_{k} \right)

Where nodes ranked higher (lower rank number) contribute more weight to the score

The final score is the average of all precision scores across the conversation. This ensures that relevant retrieval context nodes appear earlier in the ranking.

Under the hybrid eval mode, step 1 is answered by Jev, a System One model, instead: one yes/no question per node, with P(yes) >= 0.5 counted as useful. The equation and the reason are unchanged. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the whole conversation, each turn carrying its role, content and ordered retrieval_context, plus the expected_outcome, and asked three questions:

Question	Type	Weight
In every assistant turn in `turns` that has a `retrieval_context`, the documents useful for answering the preceding user message towards `expected_outcome` are listed before the documents that are not useful.	Noul	2
In every assistant turn in `turns` that has a `retrieval_context`, the first document is useful for answering the preceding user message.	Noul	1
Across the assistant turns in `turns`, how well are the `retrieval_context` documents ordered, with the ones useful for answering the preceding user message first? (Useful documents are all at the end → Useful documents are all first)	Score	1

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs

When should I use TurnContextualPrecisionMetric instead of the single-turn version?

When retrieval runs per turn in a ConversationalTestCase and you want to grade each turn's node ranking against the expected_outcome. The single-turn ContextualPrecisionMetric grades one retrieval only.

How is precision different from recall and relevancy?

Precision is ranking — are relevant nodes ranked above irrelevant ones, weighted so earlier nodes matter more. Recall measures completeness and relevancy measures signal vs noise.

The right info is in my retrieval context but precision is low — what's wrong?

Relevant nodes are ranked too low. Since the score weights nodes by position, burying a useful node beneath irrelevant ones tanks precision even when it's present. Add or tune a reranker.

Why does this metric need expected outcome?

It uses the expected_outcome to decide which nodes in the retrieval_context were useful before scoring their ranking. Without it there's no ground truth for labeling nodes relevant or irrelevant.

Required Arguments

Usage

As a standalone

How Is It Calculated?

LLM-as-a-judge

Hybrid

Jev-as-a-judge

FAQs

On this page