Plan Quality

LLM-as-a-judge

Single-turn

Referenceless

Agent

Multimodal

The Plan Quality metric is an agentic metric that extracts the task and plan from your agent's trace which are then used to evaluate the quality of the plan for completing the task. It is a self-explaining eval, which means it outputs a reason for its metric score.

info

Plan Quality metric analyzes your agent's full trace to extract the plan and evaluates that plan's quality, this requires setting up tracing.

Usage

To begin, set up tracing and simply supply the PlanQualityMetric() to your agent's @observe tag or in the evals_iterator method.

from somewhere import llm
from deepeval.tracing import observe, update_current_trace
from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import PlanQualityMetric
from deepeval.test_case import ToolCall


@observe
def tool_call(input):
    ...
    return [ToolCall(name="CheckWhether")]

@observe
def agent(input):
    tools = tool_call(input)
    output = llm(input, tools)
    update_current_trace(
        input=input,
        output=output,
        tools_called=tools
    )
    return output


# Create dataset
dataset = EvaluationDataset(goldens=[Golden(input="What's the weather like in SF?")])

# Initialize metric
metric = PlanQualityMetric(threshold=0.7, model="gpt-4o")

# Loop through dataset
for golden in dataset.evals_iterator(metrics=[metric]):
    agent(golden.input)

There are SEVEN optional parameters when creating a PlanQualityMetric:

• [Optional] threshold: a float representing the minimum passing threshold, defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to 'gpt-4o'.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.

To learn more about how the evals_iterator work, click here.

info

The PlanQualityMetric is an agentic trace-only metric, so unlike other deepeval metrics, it cannot be used as a standaolne and MUST be used in the evals_iterator or observe decorator.

How Is It Calculated?

The PlanQualityMetric score is calculated using the following steps:

• Extract Task from the trace, this defines the user's goal or intent for the agent and is actionable.
• Extract Plan from the trace, a plan is extracted from the agent's thinking or reasoning. If there are no statements that clearly define or imply a plan from the trace, the metric passes by default with a score of 1.

$$\text{Plan Quality Score} = \text{AlignmentScore}(\text{Task}, \text{Plan})$$

• The Alignment Score uses an LLM to generate the final score with all the pre-processed and extracted information like plan and task.