Faithfulness

The faithfulness metric uses LLM-as-a-judge to measure the quality of your RAG pipeline's generator by evaluating whether the actual_output factually aligns with the contents of your retrieval_context. deepeval's faithfulness metric is a self-explaining LLM-Eval, meaning it outputs a reason for its metric score.

Required Arguments

To use the FaithfulnessMetric, you'll have to provide the following arguments when creating an LLMTestCase:

• input
• actual_output
• retrieval_context

Read the How Is It Calculated section below to learn how test case parameters are used for metric calculation.

Usage

The FaithfulnessMetric() can be used for end-to-end evaluation of text-based and multimodal test cases:

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = FaithfulnessMetric(
    threshold=0.7,
    model="gpt-4.1",
    include_reason=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    retrieval_context=retrieval_context
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

from deepeval import evaluate
from deepeval.test_case import LLMTestCase, MLLMImage
from deepeval.metrics import FaithfulnessMetric

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = [
    f"The Eiffel Tower {MLLMImage(...)} is a wrought-iron lattice tower built in the late 19th century.",
    f"...",
]

metric = FaithfulnessMetric(
    threshold=0.7,
    model="gpt-4.1",
    include_reason=True
)
test_case = LLMTestCase(
    input=f"Tell me about this landmark in France: {MLLMImage(...)}",
    actual_output=f"This appears to be Eiffel Tower, which is a famous landmark in France"
    retrieval_context=retrieval_context
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

There are EIGHT optional parameters when creating a FaithfulnessMetric:

• [Optional] threshold: a float representing the minimum passing threshold, defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] truths_extraction_limit: an int which when set, determines the maximum number of factual truths to extract from the retrieval_context. The truths extracted will be used to determine the degree of factual alignment, and will be ordered by importance, decided by your evaluation model. Defaulted to None.
• [Optional] penalize_ambiguous_claims: a boolean which when set to True, will not count claims that are ambigious as faithful. Defaulted to False.
• [Optional] evaluation_template: a class of type FaithfulnessTemplate, which allows you to override the default prompts used to compute the FaithfulnessMetric score. Defaulted to deepeval's FaithfulnessTemplate.

Within components

You can also run the FaithfulnessMetric within nested components for component-level evaluation.

from deepeval.dataset import Golden
from deepeval.tracing import observe, update_current_span
...

@observe(metrics=[metric])
def inner_component():
    # Set test case at runtime
    test_case = LLMTestCase(input="...", actual_output="...")
    update_current_span(test_case=test_case)
    return

@observe
def llm_app(input: str):
    # Component can be anything from an LLM call, retrieval, agent, tool use, etc.
    inner_component()
    return

evaluate(observed_callback=llm_app, goldens=[Golden(input="Hi!")])

As a standalone

You can also run the FaithfulnessMetric on a single test case as a standalone, one-off execution.

...

metric.measure(test_case)
print(metric.score, metric.reason)

How Is It Calculated?

The FaithfulnessMetric score is calculated according to the following equation:

The FaithfulnessMetric first uses an LLM to extract all claims made in the actual_output, before using the same LLM to classify whether each claim is truthful based on the facts presented in the retrieval_context.

A claim is considered truthful if it does not contradict any facts presented in the retrieval_context.

Customize Your Template

Since deepeval's FaithfulnessMetric is evaluated by LLM-as-a-judge, you can likely improve your metric accuracy by overriding deepeval's default prompt templates. This is especially helpful if:

• You're using a custom evaluation LLM, especially for smaller models that have weaker instruction following capabilities.
• You want to customize the examples used in the default FaithfulnessTemplate to better align with your expectations.

Here's a quick example of how you can override the process of extracting claims in the FaithfulnessMetric algorithm:

from deepeval.metrics import FaithfulnessMetric
from deepeval.metrics.faithfulness import FaithfulnessTemplate

# Define custom template
class CustomTemplate(FaithfulnessTemplate):
    @staticmethod
    def generate_claims(actual_output: str):
        return f"""Based on the given text, please extract a comprehensive list of facts that can inferred from the provided text.

Example:
Example Text:
"CNN claims that the sun is 3 times smaller than earth."

Example JSON:
{{
    "claims": []
}}
===== END OF EXAMPLE ======

Text:
{actual_output}

JSON:
"""

# Inject custom template to metric
metric = FaithfulnessMetric(evaluation_template=CustomTemplate)
metric.measure(...)