Task Completion

The task completion metric uses LLM-as-a-judge to evaluate how effectively an LLM agent accomplishes a task. Task Completion is a self-explaining LLM-Eval, meaning it outputs a reason for its metric score.

Usage

First, set the eval mode:

deepeval set-eval-mode llm         # LLM-as-a-judge (default)
deepeval set-eval-mode hybrid      # LLM extracts, Jev decides
deepeval set-eval-mode system_one  # Jev-as-a-judge, no LLM

To begin, set up tracing and simply supply the TaskCompletionMetric() to your agent's @observe tag.

from deepeval.dataset import Golden, EvaluationDataset
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe

@observe()
def trip_planner_agent(input):
    destination = "Paris"
    days = 2

    @observe()
    def restaurant_finder(city):
        return ["Le Jules Verne", "Angelina Paris", "Septime"]

    @observe()
    def itinerary_generator(destination, days):
        return ["Eiffel Tower", "Louvre Museum", "Montmartre"][:days]

    itinerary = itinerary_generator(destination, days)
    restaurants = restaurant_finder(destination)

    return itinerary + restaurants


# Create dataset
dataset = EvaluationDataset(goldens=[Golden(input="This is a test query")])

# Initialize metric
task_completion = TaskCompletionMetric(threshold=0.7, model="gpt-4o")

# Loop through dataset
for golden in dataset.evals_iterator(metrics=[task_completion]):
    trip_planner_agent(golden.input)

There are TEN optional parameters when creating a TaskCompletionMetric:

• [Optional] threshold: a number representing the minimum passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
• [Optional] task: a string representing the task to be completed. If no task is supplied, it is automatically inferred from the trace. Defaulted to the None
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.

• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.

• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
• [Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Only used under hybrid or system_one eval_mode. Defaulted to jev-latest.
• [Optional] eval_mode: llm, hybrid or system_one, choosing whether an LLM, Jev, or both judge. Defaulted to the configured eval mode (llm unless set).

To learn more about how the evals_iterator work, click here.

How Is It Calculated?

You can change how the TaskCompletionMetric is calculated by setting the eval mode.

LLM-as-a-judge

The TaskCompletionMetric score is calculated according to the following equation:

• Task and Outcome are extracted from the trace (or test case for end-to-end) using an LLM.
• The Alignment Score measures how well the outcome aligns with the extracted (or user-provided) task, as judged by an LLM.

Hybrid

Under the hybrid eval mode, the LLM still extracts the task and outcome, but the alignment score is given by Jev, a System One model, as a rating from "Not achieved" to "Fully achieved" mapped onto 0 to 1. The reason states Jev's score and confidence. If a Jev call fails, the LLM makes that decision instead.

Jev-as-a-judge

Under the system_one eval mode, Jev judges the whole metric in one request. It is sent the trace (only each span's name, type, inputs, outputs and tool calls), or the raw input, actual_output and tools_called when there is no trace, plus the task if you provided one, and asked three questions:

Each answer becomes a value in $[0, 1]$ and the score is their weighted mean. No LLM is called: the reason lists each answer with its probability, and metric.confidence reports how decisive Jev was.

FAQs