Plan Quality

The Plan Quality metric is an agentic metric that extracts the task and plan from your agent's trace which are then used to evaluate the quality of the plan for completing the task. It is a self-explaining eval, which means it outputs a reason for its metric score.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

To begin, set up tracing and simply supply the PlanQualityMetric() to your agent's @observe tag or in the evals_iterator method.

from somewhere import llm
from deepeval.tracing import observe, update_current_trace
from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import PlanQualityMetric
from deepeval.test_case import ToolCall


@observe
def tool_call(input):
    ...
    return [ToolCall(name="CheckWhether")]

@observe
def agent(input):
    tools = tool_call(input)
    output = llm(input, tools)
    update_current_trace(
        input=input,
        output=output,
        tools_called=tools
    )
    return output


# Create dataset
dataset = EvaluationDataset(goldens=[Golden(input="What's the weather like in SF?")])

# Initialize metric
metric = PlanQualityMetric(threshold=0.7, model="gpt-4o")

# Loop through dataset
for golden in dataset.evals_iterator(metrics=[metric]):
    agent(golden.input)

There are NINE optional parameters when creating a PlanQualityMetric:

• [Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.

• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.

• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
• [Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
• [Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

To learn more about how the evals_iterator work, click here.

How Is It Calculated?

You can change how the PlanQualityMetric is calculated by setting the eval mode.

LLM-as-a-judge

The PlanQualityMetric score is calculated using the following steps:

• Extract Task from the trace, this defines the user's goal or intent for the agent and is actionable.
• Extract Plan from the trace, a plan is extracted from the agent's thinking or reasoning. If there are no statements that clearly define or imply a plan from the trace, the metric passes by default with a score of 1.

• The Alignment Score uses an LLM to generate the final score with all the pre-processed and extracted information like plan and task.

Hybrid

Under the hybrid eval mode, the LLM still extracts the task and plan, but the plan quality score is given by Jev, a System One model, as a five-level rating mapped onto 0 to 1. The reason states Jev's score and confidence. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the trace (only each span's name, type, inputs, outputs and tool calls) and asked three choice questions. Each has a "The agent states no plan" option that is not applicable, so an agent with no plan scores 1. The questions are:

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs