Tool Use

The Tool Use metric is a multi-turn agentic metric that evaluates your LLM agent's tool selection and argument generation capabilities. It is a self-explaining eval, which means it outputs a reason for its metric score.

Required Arguments

To use the ToolUseMetric, you'll have to provide the following arguments when creating a ConversationalTestCase:

• turns

You can learn more about how it is calculated here.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The ToolUseMetric() can be used for end-to-end multi-turn evaluations of agents.

from deepeval.test_case import Turn, ConversationalTestCase, ToolCall
from deepeval.metrics import ToolUseMetric
from deepeval import evaluate

convo_test_case = ConversationalTestCase(
    turns=[
        Turn(role="...", content="..."), 
        Turn(role="...", content="...", tools_called=[...])
    ],
)
metric = ToolUseMetric(threshold=0.5)

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[metric])

There is ONE mandatory and NINE optional parameters when creating a ToolUseMetric:

• available_tools: a list of ToolCalls that give context on all the tools that were available to your LLM agent. This list is used to evaluate your agent's tool selection capability.
• [Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.

• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.

• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
• [Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
• [Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

As a standalone

You can also run the ToolUseMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)

How Is It Calculated

You can change how the ToolUseMetric is calculated by setting the eval mode.

LLM-as-a-judge

The ToolUseMetric score is determined through the following process:

1. Compute the Tool Selection Score for each unit interaction.
2. Compute the Argument Correctness Score for all unit interactions that include tool calls.

• The Tool Selection Score evaluates whether the agent chose the most appropriate tool for the task among all the available tools.
• The Argument Correctness Score assesses whether the arguments provided in the tool call were accurate and suitable for the task. This score is only considered when a tool call has been made.

Hybrid

Under the hybrid eval mode, the tool selection and argument correctness scores of each interaction are rated by Jev, a System One model, on a five-level scale mapped onto 0 to 1, with a short line stating Jev's score and confidence as each score's reason. The equation and the LLM-written final reasons are unchanged. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the whole conversation (with the tools called in each turn) and your available_tools and asked two questions:

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs