Pattern Match

Q: Why does my pattern fail when it clearly matches part of the output?

It uses re.fullmatch, so the pattern must match the entire , not a substring. \d+ fails on "The code is 1234"; wrap with .* (e.g. .*\d+.*) to match a fragment.

Q: Does the Pattern Match metric call an LLM or cost money?

No. The PatternMatchMetric uses pure regex matching — no model, no API key, zero token cost, fully deterministic.

Q: Can I make the pattern matching case-insensitive?

Yes. Set to ignore casing when matching against the pattern. Defaults to .

Q: When should I use Pattern Match instead of Exact Match?

When the output must follow a shape rather than a fixed string — emails, dates, phone numbers, IDs. For a single literal answer, use Exact Match.

Single-turn

Referenceless

The Pattern Match metric measures whether your LLM application's actual_output matches a given regular expression pattern. This is useful for testing your model's ability to produce outputs in a specific format, structure, or syntax.

Required Arguments

To use the PatternMatchMetric, you'll have to provide the following arguments when creating an LLMTestCase:

input
actual_output

Read the How Is It Calculated section below to learn how test case parameters are used for metric calculation.

Usage

from deepeval.metrics import PatternMatchMetric
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

# Pattern: expects a valid email format
metric = PatternMatchMetric(
    pattern=r"^[\w\.-]+@[\w\.-]+\.\w+$",
    ignore_case=False,
    threshold=1.0,
    verbose_mode=True
)

test_case = LLMTestCase(
    input="Generate a valid email address.",
    actual_output="example.user@domain.com"
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

There is ONE mandatory and FOUR optional parameters when creating a PatternMatchMetric:

pattern: a string representing the regular expression pattern that the actual_output must match.
[Optional] ignore_case: a boolean which when set to True, performs case-sensitive pattern matching. Defaulted to False.
[Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 1.0.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.

As a Standalone

You can also run the PatternMatchMetric on a single test case as a standalone, one-off execution.

...

metric.measure(test_case)
print(metric.score, metric.reason)

How Is It Calculated?

The PatternMatchMetric score is calculated according to the following equation:

\text{Pattern Match Score} = \begin{cases} 1 & \text{if actual output fully matches the regex pattern}, \\ 0 & \text{otherwise} \end{cases}

The match is determined using Python's built-in regular expression engine re.fullmatch, which ensures the actual_output matches the provided pattern.

FAQs

Why does my pattern fail when it clearly matches part of the output?

It uses re.fullmatch, so the pattern must match the entire actual_output, not a substring. \d+ fails on "The code is 1234"; wrap with .* (e.g. .*\d+.*) to match a fragment.

Does the Pattern Match metric call an LLM or cost money?

No. The PatternMatchMetric uses pure regex matching — no model, no API key, zero token cost, fully deterministic.

Can I make the pattern matching case-insensitive?

Yes. Set ignore_case=True to ignore casing when matching actual_output against the pattern. Defaults to False.

When should I use Pattern Match instead of Exact Match?

When the output must follow a shape rather than a fixed string — emails, dates, phone numbers, IDs. For a single literal answer, use Exact Match.

Required Arguments

Usage

As a Standalone

How Is It Calculated?

FAQs

On this page