Image Coherence

Q: What's the difference between Image Coherence and Image Helpfulness?

ImageCoherence measures alignment — whether an image fits and flows with the text around it — while ImageHelpfulness measures usefulness. An image can be coherent yet add little value, so track both together.

Q: How is the score computed across multiple images?

Each MLLMImage is scored from the text directly above and below it, and the final score is the average (O = (ΣC_i) / n). Use max_context_size to cap the surrounding text per image.

LLM-as-a-judge

Single-turn

Multimodal

The Image Coherence metric assesses the coherent alignment of images with their accompanying text, evaluating how effectively the visual content complements and enhances the textual narrative. deepeval's Image Coherence metric is a self-explaining MLLM-Eval, meaning it outputs a reason for its metric score.

Required Arguments

To use the ImageCoherence, you'll have to provide the following arguments when creating a LLMTestCase:

input
actual_output

The input and actual_output are required to create an LLMTestCase (and hence required by all metrics) even though they might not be used for metric calculation. Read the How Is It Calculated section below to learn more.

Usage

from deepeval import evaluate
from deepeval.metrics import ImageCoherenceMetric
from deepeval.test_case import LLMTestCase, MLLMImage

metric = ImageCoherenceMetric(
    threshold=0.7,
    include_reason=True,
)
m_test_case = LLMTestCase(
    input=f"Provide step-by-step instructions on how to fold a paper airplane.",
    actual_output=f"""
        1. Take the sheet of paper and fold it lengthwise:
        {MLLMImage(url="./paper_plane_1", local=True)}
        2. Unfold the paper. Fold the top left and right corners towards the center.
        {MLLMImage(url="./paper_plane_2", local=True)}
        ...
    """
)


evaluate(test_cases=[m_test_case], metrics=[metric])

There are SIX optional parameters when creating a ImageCoherence:

[Optional] threshold: a float representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] max_context_size: a number representing the maximum number of characters in each context, as outlined in the How Is It Calculated section. Defaulted to None.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.

As a standalone

You can also run the ImageCoherenceMetric on a single test case as a standalone, one-off execution.

...

metric.measure(m_test_case)
print(metric.score, metric.reason)

How Is It Calculated?

The ImageCoherence score is calculated as follows:

Individual Image Coherence: Each image's coherence score is based on the text directly above and below the image, limited by a max_context_size in characters. If max_context_size is not supplied, all available text is used. The equation can be expressed as:

C_i = f(\text{Context}_{\text{above}}, \text{Context}_{\text{below}}, \text{Image}_i)

Final Score: The overall ImageCoherence score is the average of all individual image coherence scores for each image:

O = \frac{\sum_{i=1}^n C_i}{n}

FAQs

What's the difference between Image Coherence and Image Helpfulness?

ImageCoherence measures alignment — whether an image fits and flows with the text around it — while ImageHelpfulness measures usefulness. An image can be coherent yet add little value, so track both together.

What exactly is being aligned in a coherence check?

The image against its accompanying text: does the visual complement the narrative with consistent subject, style, and context? A coherent image looks like it belongs where it was placed.

How is the score computed across multiple images?

Each MLLMImage is scored from the text directly above and below it, and the final score is the average (O = (ΣC_i) / n). Use max_context_size to cap the surrounding text per image.

What typically causes a low Image Coherence score?

Images that clash with their context — inconsistent art style between steps, a visual depicting something other than the adjacent text, or one placed where it doesn't belong. Check metric.reason to see which image broke the flow.

Required Arguments

Usage

As a standalone

How Is It Calculated?

FAQs

On this page