Turn Relevancy

Q: When should I use TurnRelevancyMetric instead of the single-turn AnswerRelevancyMetric?

On a ConversationalTestCase when each assistant reply must stay relevant given prior turns. The single-turn AnswerRelevancyMetric judges one input/output pair in isolation.

Q: Which turns actually get scored?

Only "assistant" turns — relevant assistant turns over total assistant turns. User and tool turns never affect the numerator.

Q: What does window size control and when should I change it?

(default 10) sets how many recent turns are bundled per judgment. Lower it when only recent exchanges matter; raise it when older context still counts.

LLM-as-a-judge

Multi-turn

Referenceless

RAG

Chatbot

Multimodal

The turn relevancy metric is a conversational metric that determines whether your LLM chatbot is able to consistently generate relevant responses throughout a conversation.

Required Arguments

To use the TurnRelevancyMetric, you'll have to provide the following arguments when creating a ConversationalTestCase:

turns

You must provide the role and content for evaluation to happen. Read the How Is It Calculated section below to learn more.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The TurnRelevancyMetric() can be used for end-to-end multi-turn evaluation:

from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import TurnRelevancyMetric
from deepeval import evaluate

convo_test_case = ConversationalTestCase(
    turns=[Turn(role="...", content="..."), Turn(role="...", content="...")]
)
metric = TurnRelevancyMetric(threshold=0.5)

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[metric])

There are TEN optional parameters when creating a TurnRelevancyMetric:

[Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
[Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] window_size: an integer which defines the size of the sliding window of turns used during evaluation. Defaulted to 10.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
[Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
[Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

As a standalone

You can also run the ContextualRelevancyMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)

How Is It Calculated?

You can change how the TurnRelevancyMetric is calculated by setting the eval mode.

LLM-as-a-judge

The TurnRelevancyMetric score is calculated according to the following equation:

\text{Conversation Relevancy} = \frac{\text{Number of Turns with Relevant Assistant Content}}{\text{Total Number of Assistant Turns}}

The TurnRelevancyMetric first constructs a sliding windows of turns for each turn, before using an LLM to determine whether the last turn in each sliding window has an "assistant" content that is relevant to the previous conversational context found in the sliding window.

Hybrid

Under the hybrid eval mode, the relevancy verdict for each sliding window is answered by Jev, a System One model, instead: one yes/no question on whether the last assistant turn is relevant to the conversation so far, with P(yes) >= 0.5 counted as relevant. The equation and the LLM-written reason are unchanged. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the whole conversation and asked three questions:

Question	Type	Weight
Every assistant turn in `turns` is relevant to the conversation so far, given the user turns before it.	Noul	2
No assistant turn in `turns` ignores the user turn it responds to or changes the subject.	Noul	1
Across `turns`, how relevant are the assistant turns to the conversation so far? (Mostly irrelevant → All relevant)	Score	1

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs

When should I use TurnRelevancyMetric instead of the single-turn AnswerRelevancyMetric?

On a ConversationalTestCase when each assistant reply must stay relevant given prior turns. The single-turn AnswerRelevancyMetric judges one input/output pair in isolation.

Which turns actually get scored?

Only "assistant" turns — relevant assistant turns over total assistant turns. User and tool turns never affect the numerator.

Each reply looks fine on its own but my score is low — why?

Relevancy is judged against a sliding window of prior turns, not just the last message. A reply that ignores context established several turns ago counts as irrelevant. Use verbose_mode to see which window dragged the score down.

What does window size control and when should I change it?

window_size (default 10) sets how many recent turns are bundled per judgment. Lower it when only recent exchanges matter; raise it when older context still counts.

On this page