AI Agent Evaluation Metrics

Q: Which metric should I use to evaluate tool selection?

Use ToolCorrectnessMetric to check whether the agent picked the right tools, and ArgumentCorrectnessMetric to check whether it passed the correct arguments. Both are component-level metrics attached to the LLM span that decides tool calls.

Q: What is the difference between PlanQualityMetric and PlanAdherenceMetric?

PlanQualityMetric evaluates whether the agent's plan is logical and complete given the task. PlanAdherenceMetric evaluates whether the agent then actually followed that plan during execution. Both are trajectory metrics that require tracing and score the complete ordered trace.

Q: How does TaskCompletionMetric work?

TaskCompletionMetric reads the complete ordered trace, extracts the user's goal, and uses an LLM judge to score whether the agent completed it. It is a trajectory metric for task-critical agents, not a black-box end-to-end metric.

AI agent evaluation metrics are purpose-built measurements that assess how well autonomous LLM systems reason, plan, execute tools, and complete tasks. They can evaluate the final black-box output, the complete ordered trajectory of reasoning and actions, or a single component span.

These metrics matter because AI agents fail in fundamentally different ways than simple LLM applications. An agent might select the right tool but pass wrong arguments. It might create a brilliant plan but fail to follow it. It might complete the task but waste resources on redundant steps. AI agent evaluation metrics give you the granularity to pinpoint exactly where things go wrong.

For a broader overview of AI agent evaluation concepts and strategies, see the AI Agent Evaluation guide. For a focused explanation of metrics that score complete ordered traces, see trajectory-based evaluation. For the bigger picture of how these metrics fit into the validation layer of an agent, see what an eval harness is.

The Three Layers of AI Agent Evaluation

AI agents consist of interconnected layers that each require distinct evaluation approaches:

Layer	What It Does	Evaluation Scope	Key Metrics
Reasoning Layer	Plans tasks, creates strategies, decides what to do	Trajectory	`PlanQualityMetric`, `PlanAdherenceMetric`
Action Layer	Selects tools, generates arguments, executes calls	Component-level	`ToolCorrectnessMetric`, `ArgumentCorrectnessMetric`
Execution Layer	Orchestrates the full loop, completes objectives	Trajectory	`TaskCompletionMetric`, `StepEfficiencyMetric`

Each metric targets a specific failure mode. Together, they provide comprehensive coverage of everything that can go wrong in an AI agent pipeline.

Reasoning Layer Metrics

The reasoning layer is where your agent analyzes tasks, formulates plans, and decides on strategies. Poor reasoning leads to cascade failures—even perfect tool execution can't save an agent with a flawed plan. The metrics in this section are trajectory metrics: they require tracing and inspect planning in the context of the complete ordered trace.

Plan Quality Metric

The PlanQualityMetric is a trajectory metric that evaluates whether the plan your agent generates is logical, complete, and efficient for accomplishing the given task. It extracts the task and plan from the complete ordered trace and uses an LLM judge to assess plan quality.

from deepeval.tracing import observe
from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import PlanQualityMetric

@observe(type="tool")
def search_flights(origin, destination, date):
    return [{"id": "FL123", "price": 450}, {"id": "FL456", "price": 380}]

@observe(type="agent")
def travel_agent(user_input):
    # Agent reasons: "I need to search for flights first, then book the cheapest"
    flights = search_flights("NYC", "Paris", "2025-03-15")
    cheapest = min(flights, key=lambda x: x["price"])
    return f"Found cheapest flight: {cheapest['id']} for ${cheapest['price']}"

# Initialize metric
plan_quality = PlanQualityMetric(threshold=0.7, model="gpt-4o")

# Evaluate agent with plan quality metric
dataset = EvaluationDataset(goldens=[Golden(input="Find me the cheapest flight to Paris")])
for golden in dataset.evals_iterator(metrics=[plan_quality]):
    travel_agent(golden.input)

When to use it: Use PlanQualityMetric when your agent explicitly reasons about how to approach a task before taking action. This is common in agents that use chain-of-thought prompting or expose their planning process.

How it's calculated:

\text{Plan Quality Score} = \text{AlignmentScore}(\text{Task}, \text{Plan})

The metric extracts the task (user's goal) and plan (agent's strategy) from the trace, then uses an LLM to score how well the plan addresses the task requirements.

→ Full Plan Quality documentation

Plan Adherence Metric

The PlanAdherenceMetric is a trajectory metric that evaluates whether your agent follows its own plan during execution. Creating a good plan is only half the battle—an agent that deviates from its strategy mid-execution undermines its own reasoning.

from deepeval.tracing import observe
from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import PlanAdherenceMetric

@observe(type="tool")
def search_flights(origin, destination, date):
    return [{"id": "FL123", "price": 450}, {"id": "FL456", "price": 380}]

@observe(type="tool")
def book_flight(flight_id):
    return {"confirmation": "CONF-789", "flight_id": flight_id}

@observe(type="agent")
def travel_agent(user_input):
    # Plan: 1) Search flights, 2) Book the cheapest one
    flights = search_flights("NYC", "Paris", "2025-03-15")
    cheapest = min(flights, key=lambda x: x["price"])
    booking = book_flight(cheapest["id"])
    return f"Booked flight {cheapest['id']}. Confirmation: {booking['confirmation']}"

# Initialize metric
plan_adherence = PlanAdherenceMetric(threshold=0.7, model="gpt-4o")

# Evaluate whether agent followed its plan
dataset = EvaluationDataset(goldens=[Golden(input="Book the cheapest flight to Paris")])
for golden in dataset.evals_iterator(metrics=[plan_adherence]):
    travel_agent(golden.input)

When to use it: Use PlanAdherenceMetric alongside PlanQualityMetric when evaluating agents with explicit planning phases. If your agent creates multi-step plans, this metric ensures it actually follows through.

How it's calculated:

\text{Plan Adherence Score} = \text{AlignmentScore}(\text{(Task, Plan)}, \text{Execution Steps})

The metric extracts the task, plan, and actual execution steps from the trace, then uses an LLM to evaluate how faithfully the agent adhered to its stated plan.

→ Full Plan Adherence documentation

Action Layer Metrics

The action layer is where your agent interacts with external systems through tool calls. This is often where things go wrong—even state-of-the-art LLMs struggle with tool selection, argument generation, and call ordering. ToolCorrectnessMetric and ArgumentCorrectnessMetric are component-level metrics: attach them to the individual LLM span that selects a tool and generates its arguments.

Tool Correctness Metric

The ToolCorrectnessMetric is a component-level metric that evaluates whether your agent selects the right tools and calls them correctly. It compares the tools selected by the LLM span against a list of expected tools.

from deepeval.tracing import observe, update_current_span
from deepeval.dataset import Golden, EvaluationDataset, get_current_golden
from deepeval.metrics import ToolCorrectnessMetric
from deepeval.test_case import LLMTestCase, ToolCall

# Initialize metric
tool_correctness = ToolCorrectnessMetric(threshold=0.7)

@observe(type="tool")
def get_weather(city):
    return {"temp": "22°C", "condition": "sunny"}

# Attach metric to the LLM component where tool decisions are made
@observe(type="llm", metrics=[tool_correctness])
def call_llm(messages):
    # LLM decides to call get_weather tool
    result = get_weather("Paris")

    # Update span with tool calling information for evaluation
    update_current_span(
        input=messages[-1]["content"],
        output=f"The weather is {result['condition']}, {result['temp']}",
        expected_tools=get_current_golden().expected_tools
    )
    return result

@observe(type="agent")
def weather_agent(user_input):
    return call_llm([{"role": "user", "content": user_input}])

# Evaluate
dataset = EvaluationDataset(goldens=[Golden(input="What's the weather in Paris?", expected_tools=[ToolCall(name="get_weather")])])
for golden in dataset.evals_iterator():
    weather_agent(golden.input)

When to use it: Use ToolCorrectnessMetric when you have deterministic expectations about which tools should be called for a given task. It's particularly valuable for testing tool selection logic and identifying unnecessary tool calls.

How it's calculated:

\text{Tool Correctness} = \frac{\text{Number of Correctly Used Tools}}{\text{Total Number of Tools Called}}

The metric supports configurable strictness:

Tool name matching (default) — considers a call correct if the tool name matches
Input parameter matching — also requires input arguments to match
Output matching — additionally requires outputs to match
Ordering consideration — optionally enforces call sequence
Exact matching — requires tools_called and expected_tools to be identical

→ Full Tool Correctness documentation

Argument Correctness Metric

The ArgumentCorrectnessMetric is a component-level metric that evaluates whether your agent generates correct arguments for each tool call. Selecting the right tool with wrong arguments is as problematic as selecting the wrong tool entirely.

from deepeval.tracing import observe, update_current_span
from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import ArgumentCorrectnessMetric
from deepeval.test_case import LLMTestCase, ToolCall

# Initialize metric
argument_correctness = ArgumentCorrectnessMetric(threshold=0.7, model="gpt-4o")

@observe(type="tool")
def search_flights(origin, destination, date):
    return [{"id": "FL123", "price": 450}, {"id": "FL456", "price": 380}]

# Attach metric to the LLM component where arguments are generated
@observe(type="llm", metrics=[argument_correctness])
def call_llm(user_input):
    # LLM generates arguments for tool call
    origin, destination, date = "NYC", "London", "2025-03-15"
    flights = search_flights(origin, destination, date)

    # Update span with tool calling details for evaluation
    update_current_span(
        input=user_input,
        output=f"Found {len(flights)} flights",
    )
    return flights

@observe(type="agent")
def flight_agent(user_input):
    return call_llm(user_input)

# Evaluate - metric checks if arguments match what input requested
dataset = EvaluationDataset(goldens=[
    Golden(input="Search for flights from NYC to London on March 15th")
])
for golden in dataset.evals_iterator():
    flight_agent(golden.input)

When to use it: Use ArgumentCorrectnessMetric when correct argument values are critical for task success. This is especially important for agents that interact with APIs, databases, or external services where incorrect arguments cause failures.

How it's calculated:

\text{Argument Correctness} = \frac{\text{Number of Correctly Generated Input Parameters}}{\text{Total Number of Tool Calls}}

Unlike ToolCorrectnessMetric, this metric is fully LLM-based and referenceless—it evaluates argument correctness based on the input context rather than comparing against expected values.

→ Full Argument Correctness documentation

Execution Layer Trajectory Metrics

The execution layer encompasses the full agent loop—reasoning, acting, observing, and iterating until task completion. These trajectory metrics require tracing and assess the complete ordered sequence of steps, rather than treating the agent as a black box.

Task Completion Metric

The TaskCompletionMetric is a trajectory metric that evaluates whether your agent successfully accomplishes the intended task from the complete ordered trace. This is the ultimate measure of agent success—did it do what the user asked?

from deepeval.tracing import observe
from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import TaskCompletionMetric

@observe(type="tool")
def search_flights(origin, destination, date):
    return [{"id": "FL123", "price": 450}, {"id": "FL456", "price": 380}]

@observe(type="tool")
def book_flight(flight_id):
    return {"confirmation": "CONF-789", "flight_id": flight_id}

@observe(type="agent")
def travel_agent(user_input):
    flights = search_flights("NYC", "LA", "2025-03-15")
    cheapest = min(flights, key=lambda x: x["price"])
    booking = book_flight(cheapest["id"])
    return f"Booked flight {cheapest['id']} for ${cheapest['price']}. Confirmation: {booking['confirmation']}"

# Initialize metric - task can be auto-inferred or explicitly provided
task_completion = TaskCompletionMetric(threshold=0.7, model="gpt-4o")

# Evaluate task completion across the complete trajectory
dataset = EvaluationDataset(goldens=[
    Golden(input="Book the cheapest flight from NYC to LA for tomorrow")
])
for golden in dataset.evals_iterator(metrics=[task_completion]):
    travel_agent(golden.input)

When to use it: Use TaskCompletionMetric as a top-level success indicator for any agent. It answers the fundamental question: did the agent accomplish its goal?

How it's calculated:

\text{Task Completion Score} = \text{AlignmentScore}(\text{Task}, \text{Outcome})

The metric extracts the task (either user-provided or inferred from the trace) and the outcome, then uses an LLM to evaluate alignment. A score of 1 means complete task fulfillment; lower scores indicate partial or failed completion.

→ Full Task Completion documentation

Step Efficiency Metric

The StepEfficiencyMetric is a trajectory metric that evaluates whether your agent completes tasks without unnecessary steps. It scores the complete ordered trace, where redundant or circuitous actions are visible.

from deepeval.tracing import observe
from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import StepEfficiencyMetric

@observe(type="tool")
def search_flights(origin, destination, date):
    return [{"id": "FL123", "price": 450}, {"id": "FL456", "price": 380}]

@observe(type="tool")
def book_flight(flight_id):
    return {"confirmation": "CONF-789"}

@observe(type="agent")
def inefficient_agent(user_input):
    # Inefficient: searches twice unnecessarily
    flights1 = search_flights("NYC", "LA", "2025-03-15")
    flights2 = search_flights("NYC", "LA", "2025-03-15")  # Redundant!
    cheapest = min(flights1, key=lambda x: x["price"])
    booking = book_flight(cheapest["id"])
    return f"Booked: {booking['confirmation']}"

# Initialize metric
step_efficiency = StepEfficiencyMetric(threshold=0.7, model="gpt-4o")

# Evaluate the trajectory - metric will penalize the redundant search_flights call
dataset = EvaluationDataset(goldens=[
    Golden(input="Book the cheapest flight from NYC to LA")
])
for golden in dataset.evals_iterator(metrics=[step_efficiency]):
    inefficient_agent(golden.input)

When to use it: Use StepEfficiencyMetric alongside TaskCompletionMetric to ensure your agent isn't just successful but also efficient. This is critical for production agents where token costs and latency matter.

How it's calculated:

\text{Step Efficiency Score} = \text{AlignmentScore}(\text{Task}, \text{Execution Steps})

The metric extracts the task and all execution steps from the trace, then uses an LLM to evaluate efficiency. It penalizes redundant tool calls, unnecessary reasoning loops, and any actions not strictly required to complete the task.

→ Full Step Efficiency documentation

Putting It All Together

Here's a complete example showing trajectory metrics across the reasoning and execution layers, plus component-level metrics on the LLM tool-calling decision:

from deepeval.tracing import observe, update_current_span
from deepeval.dataset import Golden, EvaluationDataset, get_current_golden
from deepeval.test_case import LLMTestCase, ToolCall
from deepeval.metrics import (
    TaskCompletionMetric,
    StepEfficiencyMetric,
    PlanQualityMetric,
    PlanAdherenceMetric,
    ToolCorrectnessMetric,
    ArgumentCorrectnessMetric
)

# Trajectory metrics (analyze the complete ordered agent trace)
task_completion = TaskCompletionMetric()
step_efficiency = StepEfficiencyMetric()
plan_quality = PlanQualityMetric()
plan_adherence = PlanAdherenceMetric()

# Component-level metrics (analyze the LLM tool-calling decision)
tool_correctness = ToolCorrectnessMetric()
argument_correctness = ArgumentCorrectnessMetric()

# Define tools
@observe(type="tool")
def search_flights(origin, destination, date):
    return [{"id": "FL123", "price": 450}, {"id": "FL456", "price": 380}]

@observe(type="tool")
def book_flight(flight_id):
    return {"confirmation": "CONF-789", "flight_id": flight_id}

# Attach component-level metrics to the LLM component
@observe(type="llm", metrics=[tool_correctness, argument_correctness])
def call_llm(user_input):
    # LLM decides to search flights then book
    origin, destination, date = "NYC", "Paris", "2025-03-18"
    flights = search_flights(origin, destination, date)
    cheapest = min(flights, key=lambda x: x["price"])
    booking = book_flight(cheapest["id"])

    # Update span with tool info for component-level evaluation
    update_current_span(
        input=user_input,
        output=f"Booked {cheapest['id']}",
        expected_tools=get_current_golden().expected_tools
    )
    return booking

@observe(type="agent")
def travel_agent(user_input):
    booking = call_llm(user_input)
    return f"Flight booked! Confirmation: {booking['confirmation']}"

# Create evaluation dataset
dataset = EvaluationDataset(goldens=[
    Golden(input="Book a flight from NYC to Paris for next Tuesday", expected_tools=[ToolCall(name="search_flights"), ToolCall(name="book_flight")])
])

# Run evaluation with trajectory metrics
for golden in dataset.evals_iterator(
    metrics=[task_completion, step_efficiency, plan_quality, plan_adherence]
):
    travel_agent(golden.input)

Choosing the Right AI Agent Evaluation Metrics

Not every agent needs every metric. Here's a decision framework:

If Your Agent...	Evaluation Scope	Prioritize These Metrics
Uses explicit planning/reasoning	Trajectory	`PlanQualityMetric`, `PlanAdherenceMetric`
Calls multiple tools	Component-level	`ToolCorrectnessMetric`, `ArgumentCorrectnessMetric`
Has complex multi-step workflows	Trajectory	`StepEfficiencyMetric`, `TaskCompletionMetric`
Runs in production (cost-sensitive)	Trajectory	`StepEfficiencyMetric`
Is task-critical (must succeed)	Trajectory	`TaskCompletionMetric`

FAQs

What metrics does DeepEval provide for AI agents?

DeepEval ships agent metrics across three layers: reasoning (PlanQualityMetric, PlanAdherenceMetric), action (ToolCorrectnessMetric, ArgumentCorrectnessMetric), and execution (TaskCompletionMetric, StepEfficiencyMetric). The reasoning and execution metrics are trajectory metrics, while the action metrics are component-level. You can also build custom metrics with GEval or DAGMetric.

Which metric should I use to evaluate tool selection?

Use ToolCorrectnessMetric to check whether the agent picked the right tools, and ArgumentCorrectnessMetric to check whether it passed the correct arguments. Both are component-level metrics attached to the LLM span that decides tool calls.

What is the difference between PlanQualityMetric and PlanAdherenceMetric?

PlanQualityMetric evaluates whether the agent's plan is logical and complete given the task. PlanAdherenceMetric evaluates whether the agent then actually followed that plan during execution. Both are trajectory metrics that require tracing and score the complete ordered trace.

How does TaskCompletionMetric work?

TaskCompletionMetric reads the complete ordered trace, extracts the user's goal, and uses an LLM judge to score whether the agent completed it. It is a trajectory metric for task-critical agents, not a black-box end-to-end metric.

Do AI agent metrics require expected outputs?

Most agent metrics are referenceless—they only need the trace. Tool-related metrics like ToolCorrectnessMetric become reference-based when you provide expected_tools on the golden, which lets the metric compare actual versus expected tool calls.

Should I use trajectory or component-level agent metrics?

Plan quality, plan adherence, task completion, and step efficiency need the complete ordered trace, so run them as trajectory metrics via evals_iterator(metrics=[...]). Tool correctness and argument correctness evaluate one tool-calling decision, so attach them component-level via @observe(metrics=[...]) on the LLM span. Use black-box end-to-end metrics when only the final input and output should matter.

Can I run agent metrics in production?

Yes. Define a metric collection on Confident AI and reference it on your @observe decorators. The platform evaluates exported traces asynchronously, so production agents are scored continuously without added latency.

Next Steps

Now that you understand the available AI agent evaluation metrics, here's where to go next:

Set up tracing — Required for all agent metrics to capture execution traces
AI Agent Evaluation Guide — Deep dive into evaluation strategies for development and production
Trajectory-based Evals — Learn how to evaluate complete ordered agent traces
End-to-end Evals — Learn how to evaluate final black-box outputs
Component-level Evals — Learn how to attach metrics to specific components

On this page