Image Helpfulness

Q: Are my generated images actually adding value to the answer?

That's what ImageHelpfulness measures — whether each image aids comprehension (adds insight, clarifies, supports the text) versus being decorative or redundant. Read metric.reason for the per-image verdict.

Q: How is Image Helpfulness different from Image Coherence and Image Reference?

Helpfulness is usefulness (does the image improve understanding?), ImageCoherence is alignment (does it fit the narrative?), and ImageReference is grounding (does the text refer to it?). An image can be coherent yet add no value.

Q: How does scoring work when the response interleaves several images?

Each MLLMImage is scored from the text directly above and below it, and the final score is the average (O = (ΣH_i) / n). Use max_context_size to limit how much neighboring text feeds each judgment.

LLM-as-a-judge

Custom

Single-turn

Multimodal

The Image Helpfulness metric assesses how effectively images contribute to a user's comprehension of the text, including providing additional insights, clarifying complex ideas, or supporting textual details. deepeval's Image Helpfulness metric is a self-explaining MLLM-Eval, meaning it outputs a reason for its metric score.

Required Arguments

To use the ImageHelpfulness, you'll have to provide the following arguments when creating a LLMTestCase:

input
actual_output

The input and actual_output are required to create an LLMTestCase (and hence required by all metrics) even though they might not be used for metric calculation. Read the How Is It Calculated section below to learn more.

Usage

from deepeval.test_case import LLMTestCase, MLLMImage
from deepeval.metrics import ImageHelpfulnessMetric
from deepeval import evaluate

metric = ImageHelpfulnessMetric(
    threshold=0.7,
    include_reason=True,
)
m_test_case = LLMTestCase(
    input=f"Provide step-by-step instructions on how to fold a paper airplane.",
    # Replace with your MLLM app output
    actual_output=f"""
        1. Take the sheet of paper and fold it lengthwise:
        {MLLMImage(url="./paper_plane_1", local=True)}
        2. Unfold the paper. Fold the top left and right corners towards the center.
        {MLLMImage(url="./paper_plane_2", local=True)}
        ...
    """
)


evaluate(test_cases=[m_test_case], metrics=[metric])

There are SIX optional parameters when creating a ImageHelpfulnessMetric:

[Optional] threshold: a float representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] max_context_size: a number representing the maximum number of characters in each context, as outlined in the How Is It Calculated section. Defaulted to None.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.

As a standalone

You can also run the ImageHelpfulnessMetric on a single test case as a standalone, one-off execution.

...

metric.measure(m_test_case)
print(metric.score, metric.reason)

How Is It Calculated?

The ImageHelpfulness score is calculated as follows:

Individual Image Helpfulness: Each image's helpfulness score is based on the text directly above and below the image, limited by a max_context_size in characters. If max_context_size is not supplied, all available text is used. The equation can be expressed as:

H_i = f(\text{Context}_{\text{above}}, \text{Context}_{\text{below}}, \text{Image}_i)

Final Score: The overall ImageHelpfulness score is the average of all individual image helpfulness scores for each image:

O = \frac{\sum_{i=1}^n H_i}{n}

FAQs

Are my generated images actually adding value to the answer?

That's what ImageHelpfulness measures — whether each image aids comprehension (adds insight, clarifies, supports the text) versus being decorative or redundant. Read metric.reason for the per-image verdict.

How is Image Helpfulness different from Image Coherence and Image Reference?

Helpfulness is usefulness (does the image improve understanding?), ImageCoherence is alignment (does it fit the narrative?), and ImageReference is grounding (does the text refer to it?). An image can be coherent yet add no value.

How does scoring work when the response interleaves several images?

Each MLLMImage is scored from the text directly above and below it, and the final score is the average (O = (ΣH_i) / n). Use max_context_size to limit how much neighboring text feeds each judgment.

What does a low Image Helpfulness score tell me?

The model is inserting images that don't advance understanding — generic stock-like visuals, repetition of the text, or images only loosely related to their step. They may be coherent and referenced, just not helpful.

Required Arguments

Usage

As a standalone

How Is It Calculated?

FAQs

On this page