Argument Correctness

The argument correctness metric is an agentic LLM metric that assesses your LLM agent's ability to generate the correct arguments for the tools it calls. It is calculated by determining whether the arguments for each tool call is correct based on the input.

Required Arguments

To use the ArgumentCorrectnessMetric, you'll have to provide the following arguments when creating an LLMTestCase:

• input
• actual_output
• tools_called

Read the How Is It Calculated section below to learn how test case parameters are used for metric calculation.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The ArgumentCorrectnessMetric() can be used for end-to-end evaluation:

from deepeval.metrics import ArgumentCorrectnessMetric
from deepeval.test_case import LLMTestCase, ToolCall
from deepeval import evaluate

metric = ArgumentCorrectnessMetric(
    threshold=0.7,
    model="gpt-4",
    include_reason=True
)
test_case = LLMTestCase(
    input="When did Trump first raise tariffs?",
    actual_output="Trump first raised tariffs in 2018 during the U.S.-China trade war.",
    tools_called=[
        ToolCall(
            name="WebSearch Tool",
            description="Tool to search for information on the web.",
            input={"search_query": "Trump first raised tariffs year"}
        ),
        ToolCall(
            name="History FunFact Tool",
            description="Tool to provide a fun fact about the topic.",
            input={"topic": "Trump tariffs"}
        )
    ]
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

There are NINE optional parameters when creating an ArgumentCorrectnessMetric:

• [Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.

• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.

• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
• [Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
• [Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

Within components

You can also run the ArgumentCorrectnessMetric within nested components for component-level evaluation.

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.tracing import observe, update_current_span
...

@observe(metrics=[metric])
def inner_component():
    # Set test case at runtime
    test_case = LLMTestCase(input="...", actual_output="...", tools_called=[...])
    update_current_span(test_case=test_case)
    return

@observe
def llm_app(input: str):
    # Component can be anything from an LLM call, retrieval, agent, tool use, etc.
    inner_component()
    return

dataset = EvaluationDataset(goldens=[Golden(input="Hi!")])
for golden in dataset.evals_iterator():
    llm_app(golden.input)

As a standalone

You can also run the ArgumentCorrectnessMetric on a single test case as a standalone, one-off execution.

...

metric.measure(test_case)
print(metric.score, metric.reason)

How Is It Calculated?

You can change how the ArgumentCorrectnessMetric is calculated by setting the eval mode.

LLM-as-a-judge

The ArgumentCorrectnessMetric score is calculated according to the following equation:

The ArgumentCorrectnessMetric assesses the correctness of the arguments (input parameters) for each tool call, based on the task outlined in the input.

Hybrid

Under the hybrid eval mode, the assessment is answered by Jev, a System One model, instead: one yes/no question per tool call, with P(yes) >= 0.5 counted as correct. The equation and the reason are unchanged. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the raw input and the structured tools_called and asked three questions:

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs