Introduction to LLM Evaluation Metrics

deepeval offers 50+ SOTA, ready-to-use metrics for you to quickly get started with. Essentially, while a test case represents the thing you're trying to measure, the metric acts as the ruler for specific criteria of interest.

Quick Summary

Almost all predefined metrics on deepeval use LLM-as-a-judge, with techniques such as QAG (question-answer-generation), DAG (deep acyclic graphs), and G-Eval. Most score test cases representing atomic interactions, while trajectory metrics score the complete ordered trace produced by an AI agent.

All of deepeval's metrics output a score between 0-1 based on its corresponding equation, as well as score reasoning. A metric is only successful if the evaluation score is equal to or greater than threshold, which is defaulted to 0.5 for all metrics.

Metrics can score your app's black-box result, an agent's complete trajectory, or an individual component. This first example runs an end-to-end evaluation by providing metrics and test cases:

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

evaluate(
    metrics=[AnswerRelevancyMetric()],
    test_cases=[LLMTestCase(input="What's `deepeval`?", actual_output="Your favorite eval framework's favorite evals framework.")]
)

If you're logged into Confident AI before running an evaluation (deepeval login or deepeval view in the CLI), you'll also get entire testing reports on the platform:

More information on everything can be found on the Confident AI evaluation docs.

Why deepeval Metrics?

Apart from the variety of metrics offered, deepeval's metrics are a step up to other implementations because they:

• Are research-backed LLM-as-as-Judge (GEval)
• One of the most used in the world (20 million+ daily evaluations)
• Make deterministic metric scores possible (when using DAGMetric)
• Are extra reliable as LLMs are only used for extremely confined tasks during evaluation to greatly reduce stochasticity and flakiness in scores
• Provide a comprehensive reason for the scores computed
• Integrated 100% with Confident AI

Create Your First Metric

Custom Metrics

deepeval provides G-Eval, a state-of-the-art LLM evaluation framework for anyone to create a custom LLM-evaluated metric using natural language. G-Eval is available for all single-turn, multi-turn, and multimodal evals.

from deepeval.test_case import LLMTestCase, SingleTurnParams
from deepeval.metrics import GEval

test_case = LLMTestCase(input="...", actual_output="...", expected_output="...")
correctness = GEval(
    name="Correctness",
    criteria="Correctness - determine if the actual output is correct according to the expected output.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
    strict_mode=True
)

correctness.measure(test_case)
print(correctness.score, correctness.reason)

from deepeval.test_case import Turn, MultiTurnParams, ConversationalTestCase
from deepeval.metrics import ConversationalGEval

convo_test_case = ConversationalTestCase(turns=[Turn(role="...", content="..."), Turn(role="...", content="...")])
professionalism_metric = ConversationalGEval(
    name="Professionalism",
    criteria="Determine whether the assistant has acted professionally based on the content."
    evaluation_params=[MultiTurnParams.CONTENT],
    strict_mode=True
)

professionalism_metric.measure(convo_test_case)
print(professionalism_metric.score, professionalism_metric.reason)

Under the hood, deepeval first generates a series of evaluation steps, before using these steps in conjunction with information in an LLMTestCase for evaluation. For more information, visit the G-Eval documentation page.

Default Metrics

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe

@observe()
def trip_planner_agent(input):

    @observe()
    def itinerary_generator(destination, days):
        return ["Eiffel Tower", "Louvre Museum", "Montmartre"][:days]

    return itinerary_generator("Paris", 2)

dataset = EvaluationDataset(goldens=[Golden(input="Plan a two-day trip to Paris")])

for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric(threshold=0.5)]):
    trip_planner_agent(golden.input)

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(input="...", actual_output="...")
relevancy = AnswerRelevancyMetric(threshold=0.5)

relevancy.measure(test_case)
print(relevancy.score, relevancy.reason)

from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import ConversationalGEval

convo_test_case = ConversationalTestCase(turns=[Turn(role="...", content="..."), Turn(role="...", content="...")])
role_adherence = RoleAdherenceMetric(threshold=0.5)

role_adherence.measure(convo_test_case)
print(role_adherence.score, role_adherence.reason)

from deepeval.test_case import LLMTestCase, MLLMImage
from deepeval.metrics import ImageCoherenceMetric

test_case = LLMTestCase(input=f"What does this image say? {MLLMImage(...)}", actual_output="No idea!")
image_coherence = ImageCoherenceMetric(threshold=0.5)

image_coherence.measure(test_case)
print(image_coherence.score, image_coherence.reason)

from deepeval.test_case import LLMTestCase
from deepeval.metrics import BiasMetric

test_case = LLMTestCase(input="...", actual_output="...")
bias = BiasMetric(threshold=0.5)

bias.measure(test_case)
print(bias.score, bias.reason)

Choosing Your Metrics

These are the metric categories to consider when choosing your metrics:

• Custom metrics are use case specific and architecture agnostic:
  • G-Eval – best for subjective criteria like correctness, coherence, or tone; easy to set up.
  • DAG – decision-tree metric for objective or mixed criteria (e.g., verify format before tone).
  • Start with G-Eval for simplicity; use DAG for more control. You can also subclass BaseMetric to create your own.
• Generic metrics are system specific and use case agnostic:
  • Agent trajectory metrics: evaluate task completion, execution efficiency, planning, and plan adherence across the complete trace
  • Agent component metrics: evaluate tool selection and argument generation at individual action steps
  • RAG metrics: measures retriever and generator separately
  • Multi-turn metrics: measure overall dialogue quality
  • Combine these for multi-component LLM systems.
• Reference vs. Referenceless:
  • Reference-based metrics need ground truth (e.g., contextual recall or tool correctness).
  • Referenceless metrics work without labeled data, ideal for online or production evaluation.
  • Check each metric’s docs for required parameters.

When deciding on metrics, no matter how tempting, try to limit yourself to no more than 5 metrics, with this breakdown:

• 2-3 generic, system-specific metrics (e.g. task completion for agents, contextual precision for RAG)
• 1-2 custom, use case-specific metrics (e.g. helpfulness for a medical chatbot, format correctness for summarization)

The goal is to force yourself to prioritize and clearly define your evaluation criteria. This will not only help you use deepeval, but also help you understand what you care most about in your LLM application.

Here are some additional ideas if you're not sure:

• AI agents: Start with TaskCompletionMetric for overall trajectory quality, add StepEfficiencyMetric, PlanAdherenceMetric, or PlanQualityMetric when the execution path matters, and use ToolCorrectnessMetric or ArgumentCorrectnessMetric to diagnose individual LLM tool-calling decisions
• RAG: Focus on the AnswerRelevancyMetric (evaluates actual_output alignment with the input) and FaithfulnessMetric (checks for hallucinations against retrieval_context)
• Chatbots: Implement a ConversationCompletenessMetric to assess overall conversation quality
• Custom Requirements: When standard metrics don't fit your needs, create custom evaluations with G-Eval or DAG frameworks

In some cases, where your LLM model is doing most of the heavy lifting, it is not uncommon to have more use case specific metrics.

Configure LLM Judges

You can use ANY LLM judge in deepeval, including OpenAI, Azure OpenAI, Ollama, Anthropic, Gemini, LiteLLM, etc. You can also wrap your own LLM API in deepeval's DeepEvalBaseLLM class to use ANY model of your choice. Click here for full guide.

export OPENAI_API_KEY=<your-openai-api-key>

%env OPENAI_API_KEY=<your-openai-api-key>

deepeval set-azure-openai \
    --base-url=<endpoint> \ # e.g. https://example-resource.azure.openai.com/
    --model=<model_name> \ # e.g. gpt-4.1
    --deployment-name=<deployment_name> \  # e.g. Test Deployment
    --api-version=<api_version> \ # e.g. 2025-01-01-preview
    --model-version=<model_version> # e.g. 2024-11-20

deepeval unset-azure-openai

deepeval set-ollama --model=deepseek-r1:1.5b

deepeval set-ollama --model=deepseek-r1:1.5b \
    --base-url="http://localhost:11434"

deepeval unset-ollama

deepeval set-gemini \
    --model=<model_name> # e.g. "gemini-2.0-flash-001"

from langchain_openai import AzureChatOpenAI
from deepeval.models.base_model import DeepEvalBaseLLM

class AzureOpenAI(DeepEvalBaseLLM):
    def __init__(
        self,
        model
    ):
        self.model = model

    def load_model(self):
        return self.model

    def generate(self, prompt: str) -> str:
        chat_model = self.load_model()
        return chat_model.invoke(prompt).content

    async def a_generate(self, prompt: str) -> str:
        chat_model = self.load_model()
        res = await chat_model.ainvoke(prompt)
        return res.content

    def get_model_name(self):
        return "Custom Azure OpenAI Model"

# Replace these with real values
custom_model = AzureChatOpenAI(
    openai_api_version=api_version,
    azure_deployment=azure_deployment,
    azure_endpoint=azure_endpoint,
    openai_api_key=openai_api_key,
)
azure_openai = AzureOpenAI(model=custom_model)
print(azure_openai.generate("Write me a joke"))

from deepeval.metrics import AnswerRelevancyMetric
...

metric = AnswerRelevancyMetric(model=azure_openai)

Using Metrics

There are four ways you can use metrics:

1. End-to-end evals, treating your LLM system as a black-box and evaluating the system inputs and outputs.
2. Trajectory-based evals, evaluating the complete ordered chain of steps taken by an AI agent.
3. Component-level evals, placing metrics on individual components in your LLM app instead.
4. One-off (or standalone) evals, where you would use a metric to execute it individually.

For End-to-End Evals

To run end-to-end evaluations of your LLM system, provide black-box metrics with a list of test cases:

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

test_case = LLMTestCase(input="...", actual_output="...")

evaluate(test_cases=[test_case], metrics=[AnswerRelevancyMetric()])

The evaluate() function or deepeval test run is the best way to run evaluations. They offer tons of features out of the box, including caching, parallelization, cost tracking, error handling, and integration with Confident AI.

For Trajectory-Based Evals

Trajectory metrics analyze relationships between the plans, model calls, tools, handoffs, and intermediate steps in an agent's complete trace. Metrics carrying the Trajectory tag—Task Completion, Step Efficiency, Plan Adherence, and Plan Quality—require LLM tracing.

Pass trajectory metrics to evals_iterator(), then invoke your traced agent once for each golden:

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric
from app import my_agent

dataset = EvaluationDataset(
    goldens=[Golden(input="Plan a three-day trip to Paris")]
)

for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    my_agent(golden.input)

Each metric receives the complete ordered trace after the agent finishes. See trajectory-based evaluation for instrumentation examples, CI/CD usage, and guidance on combining trajectory and component metrics.

For Component-Level Evals

To run component-level evaluations of your LLM system using any metric of your choice, simply decorate your components with @observe and create test cases at runtime:

from deepeval.tracing import observe, update_current_span
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import AnswerRelevancyMetric

# 1. observe() decorator traces LLM components
@observe()
def llm_app(input: str):
    # 2. Supply metric at any component
    @observe(metrics=[AnswerRelevancyMetric()])
    def nested_component():
        # 3. Create test case at runtime
        update_current_span(test_case=LLMTestCase(...))
        pass

    nested_component()

# 4. Create dataset
dataset = EvaluationDataset(goldens=[Golden(input="Test input")])

# 5. Loop through dataset
for goldens in dataset.evals_iterator():
    # Call LLM app
    llm_app(golden.input)

Use Metrics with Integrations

deepeval's framework integrations capture the traces and spans that metrics evaluate, so you can score complete agent trajectories or individual framework-emitted components without recreating the execution structure yourself.

Here are a few examples available for your selected SDK:

LangChain

Evaluate complete agent runs, model calls, tools, and retrievers.

Pydantic AI

Evaluate Pydantic AI agents and their instrumented spans.

AWS AgentCore

Evaluate AgentCore-hosted agents and their nested spans.

Google ADK

Evaluate ADK agent runs, tools, and model calls.

Pass metrics to evals_iterator() for end-to-end or trajectory-based evaluation. Attach or stage metrics on framework-emitted spans for component-level evaluation. Each integration guide shows the supported patterns for that framework.

Browse all framework integrations →

For One-Off Evals

You can also execute each metric individually. All metrics in deepeval, including custom metrics that you create:

• can be executed via the metric.measure() method
• can have its score accessed via metric.score, which ranges from 0 - 1
• can have its score reason accessed via metric.reason
• can have its status accessed via metric.is_successful()
• can be used to evaluate test cases or entire datasets, with or without Pytest
• has a threshold that acts as the threshold for success. metric.is_successful() is only true if metric.score is above/below threshold
• has a flaky property, which when turned on stops the metric's verdict from deciding a test case's pass/fail status
• has a strict_mode property, which when turned on enforces metric.score to a binary one
• has a verbose_mode property, which when turned on prints metric logs whenever a metric is executed

In addition, all metrics in deepeval execute asynchronously by default.

You can configure this behavior using the async_mode parameter when instantiating a metric.

Here's a quick example:

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

# Initialize a test case
test_case = LLMTestCase(...)

# Initialize metric with threshold
metric = AnswerRelevancyMetric(threshold=0.5)
metric.measure(test_case)

print(metric.score, metric.reason)

All of deepeval's metrics give a reason alongside its score.

Using Metrics Async

When a metric's async_mode=True (which is the default for all metrics), invocations of metric.measure() will execute internal algorithms concurrently. However, it's important to note that while operations INSIDE measure() execute concurrently, the metric.measure() call itself still blocks the main thread.

To measure multiple metrics at once and NOT block the main thread, use the asynchronous a_measure() method instead.

import asyncio

...

# Remember to use async
async def long_running_function():
    # These will all run at the same time
    await asyncio.gather(
        metric1.a_measure(test_case),
        metric2.a_measure(test_case),
        metric3.a_measure(test_case),
        metric4.a_measure(test_case)
    )
    print("Metrics finished!")

asyncio.run(long_running_function())

Debug A Metric Judgement

You can turn on verbose_mode for ANY deepeval metric at metric initialization to debug a metric whenever the measure() or a_measure() method is called:

...

metric = AnswerRelevancyMetric(verbose_mode=True)
metric.measure(test_case)

Customize Metric Prompts

All of deepeval's metrics use LLM-as-a-judge evaluation with unique default prompt templates for each metric. While deepeval has well-designed algorithms for each metric, you can customize these prompt templates to improve evaluation accuracy and stability. Simply provide a custom template class as the evaluation_template parameter to your metric of choice (example below).

You'll find this particularly valuable when using a custom LLM, as deepeval's default metrics are optimized for OpenAI's models, which are generally more powerful than most custom LLMs.

Here's a quick example of how you can define a custom AnswerRelevancyTemplate and inject it into the AnswerRelevancyMetric through the evaluation_params parameter:

from deepeval.metrics.answer_relevancy import AnswerRelevancyTemplate
from deepeval.metrics import AnswerRelevancyMetric

# Define custom template
class CustomTemplate(AnswerRelevancyTemplate):
    @staticmethod
    def generate_statements(actual_output: str):
        return f"""Given the text, breakdown and generate a list of statements presented.

Example:
Our new laptop model features a high-resolution Retina display for crystal-clear visuals.

{{
    "statements": [
        "The new laptop model has a high-resolution Retina display."
    ]
}}
===== END OF EXAMPLE ======

Text:
{actual_output}

JSON:
"""

# Inject custom template to metric
metric = AnswerRelevancyMetric(evaluation_template=CustomTemplate)
metric.measure(...)

Metric Thresholds

Every metric accepts a threshold parameter (defaulted to 0.5) that decides whether it passes or fails: metric.is_successful() is only True if metric.score reaches the threshold. Every metric scores in the same direction: 1 is a pass, 0 is a failure, and threshold is always a minimum.

from deepeval.metrics import AnswerRelevancyMetric

metric = AnswerRelevancyMetric(threshold=0.7)

You can also set threshold=None to run a metric in score-only mode. The score and reason are still computed, recorded, and reported, but the metric has no pass/fail opinion — metric.is_successful() returns None, and the metric never contributes to its test case's pass/fail status.

metric = AnswerRelevancyMetric(threshold=None)

Flaky Metrics

Every metric also accepts a flaky parameter (defaulted to False). A flaky metric behaves exactly like a normal one — its score, reason, and verdict are all computed and reported — but its failure never decides its test case's pass/fail status.

from deepeval.metrics import AnswerRelevancyMetric

metric = AnswerRelevancyMetric(threshold=0.7, flaky=True)

This is useful for metrics you know are noisy — for example ones with borderline scores that flip between passing and failing across runs — that you still want to keep measuring and tracking without letting them gate deployments. Flaky pass/fail counts are shown separately in test run results, and the flaky status is also logged on Confident AI.

What About Non-LLM-as-a-judge Metrics?

If you're looking to use something like ROUGE, BLEU, or BLEURT, etc. you can create a custom metric and use the scorer module available in deepeval for scoring by following this guide.

The scorer module is available but not documented because our experience tells us these scorers are not useful as LLM metrics where outputs require a high level of reasoning to evaluate.

FAQs