Role Adherence

Q: How is this different from the single-turn Role Violation metric?

Role Violation inspects one output in isolation, whereas RoleAdherenceMetric walks every assistant turn of a ConversationalTestCase and measures how consistently the bot held its chatbot_role.

Q: Which turns are scored, and how?

Only assistant turns — those that stayed in chatbot_role over total assistant turns, with each turn judged against the preceding conversation.

Q: What should chatbot_role contain?

The persona you expect the bot to maintain (for example, "a 1920s detective" or "a formal banking assistant"). It's the reference every assistant turn is checked against, so make it specific.

LLM-as-a-judge

Multi-turn

Referenceless

Chatbot

Multimodal

The role adherence metric is a conversational metric that determines whether your LLM chatbot is able to adhere to its given role throughout a conversation.

Required Arguments

To use the RoleAdherenceMetric, you'll have to provide the following arguments when creating a ConversationalTestCase:

turns
chatbot_role

You must provide the role and content for evaluation to happen. Read the How Is It Calculated section below to learn more.

Usage

The RoleAdherenceMetric() can be used for end-to-end multi-turn evaluation:

from deepeval import evaluate
from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import RoleAdherenceMetric

convo_test_case = ConversationalTestCase(
    chatbot_role="...",
    turns=[Turn(role="...", content="..."), Turn(role="...", content="...")]
)
metric = RoleAdherenceMetric(threshold=0.5)

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[metric])

There are SIX optional parameters when creating a RoleAdherenceMetric:

[Optional] threshold: a float representing the minimum passing threshold, defaulted to 0.5.
[Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
[Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.

As a standalone

You can also run the RoleAdherenceMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)

How Is It Calculated?

The RoleAdherenceMetric score is calculated according to the following equation:

\text{Role Adherence} = \frac{\text{Number of Assistant Turns that Adhered to Chatbot Role in Conversation}}{\text{Total Number of Assistant Turns in Conversation}}

The RoleAdherenceMetric iterates over each assistant turn and uses an LLM to evaluate whether the content adheres to the specified chatbot_role, using previous conversation turns as context.

FAQs

Why does my bot break character partway through a conversation?

Drift creeps in on later turns as the conversation grows or the user pushes back. RoleAdherenceMetric scores each assistant turn against your chatbot_role using prior turns as context, so the reason output flags which turn broke character.

How is this different from the single-turn Role Violation metric?

Role Violation inspects one output in isolation, whereas RoleAdherenceMetric walks every assistant turn of a ConversationalTestCase and measures how consistently the bot held its chatbot_role.

Which turns are scored, and how?

Only assistant turns — those that stayed in chatbot_role over total assistant turns, with each turn judged against the preceding conversation.

What should chatbot_role contain?

The persona you expect the bot to maintain (for example, "a 1920s detective" or "a formal banking assistant"). It's the reference every assistant turn is checked against, so make it specific.

Required Arguments

Usage

As a standalone

How Is It Calculated?

FAQs

On this page