Skip to main content

Plan Quality

LLM-as-a-judge
Single-turn
Referenceless
Agent

The Plan Quality metric is an agentic metric that extracts the task and plan from your agent's trace which are then used to evaluate the quality of the plan for completing the task. It is a self-explaining eval, which means it outputs a reason for its metric score.

info

Plan Quality metric analyzes your agent's full trace to extract the plan and evaluates that plan's quality, this requires setting up tracing.

Usage​

To begin, set up tracing and simply supply the PlanQualityMetric() to your agent's @observe tag or in the evals_iterator method.

from somewhere import llm
from deepeval.tracing import observe
from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import PlanQualityMetric


@observe
def tool_call(input):
    ...
    return [ToolCall(name="CheckWhether")]

@observe
def agent(input):
    tools = tool_call(input)
    output = llm(input, tools)
    update_current_trace(
        input=input,
        output=output,
        tools_called=tools
    )
    return output


# Create dataset
dataset = EvaluationDataset(goldens=[Golden(input="What's the weather like in SF?")])

# Initialize metric
plan_quality = PlanQualityMetric(threshold=0.7, model="gpt-4o")

# Loop through dataset
for goldens in dataset.evals_iterator(metrics=[plan_quality]):
    trip_planner_agent(golden.input)

There are SEVEN optional parameters when creating a PlanQualityMetric:

  • [Optional] threshold: a float representing the minimum passing threshold, defaulted to 0.5.
  • [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to 'gpt-4o'.
  • [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
  • [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
  • [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
  • [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.

To learn more about how the evals_iterator work, click here.

As a standalone​

You can also run the PlanQualityMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)
caution

This is great for debugging or if you wish to build your own evaluation pipeline, but you will NOT get the benefits (testing reports) and all the optimizations (speed, caching, computation) the evaluate() function or deepeval test run offers.

How Is It Calculated?​

The PlanAdherenceMetric score is calculated using the following steps:

  • Extract Task from the trace, this defines the user's goal or intent for the agent and is actionable.
  • Extract Plan from the trace, a plan is extracted from the agent's thinking or reasoning. If there are no statements that clearly define or imply a plan from the trace, the metric passes by default with a score of 1.
Plan Adherence Score=AlignmentScore(Task,Plan)\text{Plan Adherence Score} = \text{AlignmentScore}(\text{Task}, \text{Plan})
  • The Alignment Score uses an LLM to generate the final score with all the pre-processed and extracted information like plan and task.