Citation Faithfulness

The citation faithfulness metric uses LLM-as-a-judge to check whether every [N] citation marker in your RAG pipeline's actual_output points to the passage in retrieval_context that actually supports the specific claim the marker is attached to. deepeval's citation faithfulness metric is a self-explaining LLM-Eval, meaning it outputs a reason for its metric score.

Required Arguments

To use the CitationFaithfulnessMetric, you'll have to provide the following arguments when creating an LLMTestCase:

• input
• actual_output
• retrieval_context

The retrieval_context passages are numbered ([1], [2], ...) before being shown to the judge, and the [N] markers in actual_output refer to those passage numbers.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The CitationFaithfulnessMetric() can be used for end-to-end evaluation of text-based test cases:

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics.community import CitationFaithfulnessMetric

# Replace this with the actual output from your LLM application.
# The completion-year claim is cited to passage [1], which only covers height.
actual_output = "The Eiffel Tower was completed in 1889 [1]."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = [
    "The Eiffel Tower stands 330 metres tall in Paris.",
    "The Eiffel Tower was completed in 1889 for the World Fair.",
]

metric = CitationFaithfulnessMetric()
test_case = LLMTestCase(
    input="How tall is the Eiffel Tower and when was it completed?",
    actual_output=actual_output,
    retrieval_context=retrieval_context,
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

In the example above the FaithfulnessMetric would pass the answer, because the completion-year claim is supported by passage [2]. The CitationFaithfulnessMetric fails it, because the [1] citation points to the height passage rather than the passage that supports the claim.

There are nine optional parameters when creating a CitationFaithfulnessMetric:

• [Optional] threshold: a float representing the minimum passing score. Can also be set to None to run the metric in score-only mode. Defaulted to 1.0. The score is 1.0 for a faithful answer and 0.0 for an unfaithful one, so the default requires a faithful verdict to pass.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 0 for perfection, 1 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console. Defaulted to False.
• [Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
• [Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
• [Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

How Is It Calculated?

You can change how the CitationFaithfulnessMetric is calculated by setting the eval mode.

LLM-as-a-judge

The CitationFaithfulnessMetric score is calculated according to the following equation:

A single LLM judge reads the question, the numbered passages, and the answer, then returns a binary verdict:

• faithful (score 1.0): every factual claim is supported by the passages, and every [N] citation marker points to a passage that supports the claim it is attached to.
• unfaithful (score 0.0): at least one claim is unsupported, contradicts a passage, or carries a citation [N] where passage N does not support that claim, even if some other passage would.

Hybrid

Under the hybrid eval mode, the faithful / unfaithful verdict is answered by Jev, a System One model, as one yes/no question, with P(yes) >= 0.5 counted as faithful. The reason states Jev's probability and confidence. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the actual_output and the numbered passages and asked three questions:

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

The metric is pass/fail: a weighted mean of 0.5 or above scores 1, anything lower scores 0.