Prompt Alignment

The prompt alignment metric uses LLM-as-a-judge to measure whether your LLM application is able to generate actual_outputs that aligns with any instructions specified in your prompt template. deepeval's prompt alignment metric is a self-explaining LLM-Eval, meaning it outputs a reason for its metric score.

Required Arguments

To use the PromptAlignmentMetric, you'll have to provide the following arguments when creating an LLMTestCase:

• input
• actual_output

Read the How Is It Calculated section below to learn how test case parameters are used for metric calculation.

Usage

The PromptAlignmentMetric() can be used for end-to-end evaluation:

from deepeval.metrics import PromptAlignmentMetric
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

metric = PromptAlignmentMetric(
    prompt_instructions=["Reply in all uppercase"],
    model="gpt-4",
    include_reason=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    # Replace this with the actual output from your LLM application
    actual_output="We offer a 30-day full refund at no extra cost."
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

There are ONE mandatory and SEVEN optional parameters when creating an PromptAlignmentMetric:

• prompt_instructions: a list of strings specifying the instructions you want followed in your prompt template.
• [Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.

• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.

• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.

Within components

You can also run the PromptAlignmentMetric within nested components for component-level evaluation.

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.tracing import observe, update_current_span
...

@observe(metrics=[metric])
def inner_component():
    # Set test case at runtime
    test_case = LLMTestCase(input="...", actual_output="...")
    update_current_span(test_case=test_case)
    return

@observe
def llm_app(input: str):
    # Component can be anything from an LLM call, retrieval, agent, tool use, etc.
    inner_component()
    return

dataset = EvaluationDataset(goldens=[Golden(input="Hi!")])
for golden in dataset.evals_iterator():
    llm_app(golden.input)

As a standalone

You can also run the PromptAlignmentMetric on a single test case as a standalone, one-off execution.

...

metric.measure(test_case)
print(metric.score, metric.reason)

How Is It Calculated?

The PromptAlignmentMetric score is calculated according to the following equation:

The PromptAlignmentMetric uses an LLM to classify whether each prompt instruction is followed in the actual_output using additional context from the input.

FAQs