MCP-Use

The MCP Use is a metric that is used to evaluate how effectively an MCP based LLM agent makes use of the mcp servers it has access to. It uses LLM-as-a-judge to evaluate the MCP primitives called as well as the arguments generated by the LLM app.

Required Arguments

To use the MCPUseMetric, you'll have to provide the following arguments when creating an LLMTestCase:

• input
• actual_output
• mcp_servers

You'll also need to supply any mcp_tools_called, mcp_resources_called, and mcp_prompts_called if used, for evaluation to happen. Click here to learn about how it is calculated.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

The MCPUseMetric can be used on a single-turn LLMTestCase case with MCP parameters. Click here to see how to create an MCP single-turn test case.

from deepeval.test_case import LLMTestCase, MCPServer
from deepeval.metrics import MCPUseMetric
from deepeval import evaluate

test_case = LLMTestCase(
    input="...", # Your input here
    actual_output="...", # Your LLM app's final output here
    mcp_servers=[MCPServer(...)] # Your MCP server's data
    # MCP primitives used (if any)
)

metric = MCPUseMetric()

# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)

evaluate([test_case], [metric])

There are NINE optional parameters when creating a MCPTaskCompletionMetric:

• [Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.

• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.

• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
• [Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
• [Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

As a standalone

You can also run the MCPUseMetric on a single test case as a standalone, one-off execution.

...

metric.measure(convo_test_case)
print(metric.score, metric.reason)

How Is It Calculated

You can change how the MCPUseMetric is calculated by setting the eval mode.

LLM-as-a-judge

The MCPUseMetric score is calculated according to the following equation:

The AlignmentScore is judged by an evaluation model based on which primitives were called and their generated arguments with respect to the user's input.

Hybrid

Under the hybrid eval mode, the primitive usage and argument correctness scores are rated by Jev, a System One model, on a four-level scale mapped onto 0 to 1, with a short line stating Jev's score and confidence as each score's reason. The equation is unchanged. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the input, actual_output, the MCP calls made and the available mcp_servers and asked three questions:

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs