Text to Image
The Text to Image metric assesses the performance of image generation tasks by evaluating the quality of synthesized images based on semantic consistency and perceptual quality. deepeval's Text to Image metric is a self-explaining MLLM-Eval, meaning it outputs a reason for its metric score.
Required Arguments
To use the TextToImageMetric, you'll have to provide the following arguments when creating a LLMTestCase:
inputactual_output
The input and actual_output are required to create an LLMTestCase (and hence required by all metrics) even though they might not be used for metric calculation. Read the How Is It Calculated section below to learn more.
Usage
from deepeval import evaluate
from deepeval.metrics import TextToImageMetric
from deepeval.test_case import LLMTestCase, MLLMImage
metric = TextToImageMetric(
threshold=0.7,
include_reason=True,
)
m_test_case = LLMTestCase(
input=f"Generate an image of a blue pair of shoes.",
# Replace with your MLLM app output
actual_output=f"{MLLMImage(url='https://shoe-images.com/edited-shoes', local=False)}",
)
evaluate(test_cases=[m_test_case], metrics=[metric])There are SIX optional parameters when creating a TextToImageMetric:
- [Optional]
threshold: a float representing the minimum passing threshold. Can also be set toNoneto run the metric in score-only mode. Defaulted to0.5. - [Optional]
include_reason: a boolean which when set toTrue, will include a reason for its evaluation score. Defaulted toTrue. - [Optional]
strict_mode: a boolean which when set toTrue, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted toFalse. - [Optional]
async_mode: a boolean which when set toTrue, enables concurrent execution within themeasure()method. Defaulted toTrue. - [Optional]
verbose_mode: a boolean which when set toTrue, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted toFalse. - [Optional]
flaky: a boolean which when set toTrue, marks the metric as flaky. Defaulted toFalse.
As a standalone
You can also run the TextToImageMetric on a single test case as a standalone, one-off execution.
...
metric.measure(m_test_case)
print(metric.score, metric.reason)How Is It Calculated?
The TextToImageMetric score is calculated according to the following equation:
The TextToImageMetric score combines Semantic Consistency (SC) and Perceptual Quality (PQ) sub-scores to provide a comprehensive evaluation of the synthesized image. The final overall score is derived by taking the square root of the product of the minimum SC and PQ scores.
SC Scores
These scores assess aspects such as alignment with the prompt and resemblance to concepts. The minimum value among these sub-scores represents the SC score. During the SC evaluation, both the input conditions and the synthesized image are used.
PQ Scores
These scores evaluate the naturalness and absence of artifacts in the image. The minimum value among these sub-scores represents the PQ score. For the PQ evaluation, only the synthesized image is used to prevent confusion from the input conditions.
FAQs
Should I use Text to Image or Image Editing to evaluate my model?
TextToImageMetric when generating from a prompt alone (no source image). Use the ImageEditingMetric when transforming an existing image.How many images should the input and output contain for this metric?
input and exactly 1 MLLMImage in the actual_output. A source image in the input means you want Image Editing instead.My Text to Image score is low — is it a prompt-alignment or an image-quality problem?
metric.reason to see which dragged the score down.Why does one weak sub-score tank the overall score so hard?
O = sqrt(min(SC) · min(PQ)), multiplying the minimum SC and PQ sub-scores. So a single failing aspect (e.g. one badly rendered object) intentionally caps the whole result.