Introduction to LLM Evals

Q: What are the three components an LLM evaluation needs?

A test case (the input and to score), one or more metrics, and optionally an evaluation dataset to run those metrics across many test cases at once.

Q: Do I need a deepeval account or API key to run evals?

No. deepeval runs entirely locally. The only key you typically need is an LLM provider key such as OPENAI_API_KEY for metrics that use an LLM judge. A Confident AI account is optional and only needed to push results to the cloud.

Q: Should I run evals in CI/CD or with the evaluate() function?

Use when you want evals gated in CI/CD, and evaluate() for scripts where you want parallel execution without the CLI. Both run the same metrics on the same test cases.

Q: How does deepeval score outputs without an exact match to the expected output?

Most of deepeval's 30+ metrics use an LLM judge with techniques like G-Eval, DAG, and QAG rather than string matching, so a correct answer phrased differently from the can still pass.

Quick Summary

Evaluation refers to the process of testing your LLM application outputs. At the top level, there are TWO types of LLM apps:

Single-turn: Each interaction is one input and one output, such as a single-step AI agent or a RAG QA app.
Multi-turn: Each interaction is a back-and-forth conversation, such as a customer support AI agent, chatbot, or voice agent.

Either type can also vary in:

Modality: Inputs and outputs can be text, images, voice, or video.
Data access: Your LLM app can pull in external data through tool calls, MCP, or RAG.

Here's what an ideal evaluation workflow looks like using deepeval, and the components it requires, for each type. Multi-turn LLM apps have an extra simulation step that generates conversations for you to evaluate:

Test cases (LLMTestCases)
Metrics
Evaluation dataset of Goldens
[Optional] Classifiers, which label each test case from a closed set instead of scoring it

Test cases (ConversationalTestCases)
Multi-turn metrics
Evaluation dataset of ConversationalGoldens
Conversation simulator, which turns each ConversationalGolden into a conversation
[Optional] Classifiers, which label each test case from a closed set instead of scoring it

There are THREE types of LLM evaluations in deepeval:

End-to-end evaluation: The overall input and outputs of your LLM system.
Trajectory-based evaluation: The complete chain of decisions and actions your agent takes, evaluated as one unit.
Component-level evaluation: The individual inner workings of your LLM system.

All three can be done using either deepeval test run in CI/CD pipelines, or via the evaluate() function in your scripts.

Evals

Evals are what score your LLM application in deepeval. There are two kinds:

Metrics, which are quantitative evals that score each test case.
Classifiers, which are categorical evals that label each test case from a closed set.

Metrics

Metrics are quantitative evals: each one returns a score between 0 and 1 along with a reason, and passes or fails against a threshold. deepeval offers 30+ metrics, most of which are evaluated using LLMs (visit the metrics section to learn why).

from deepeval.metrics import AnswerRelevancyMetric

answer_relevancy_metric = AnswerRelevancyMetric()

Classifiers

Classifiers are categorical evals: instead of a score, they label each test case with one option from a closed set, such as whether your LLM app refused a request. Visit the classifiers section to learn when to use them over metrics.

from deepeval.classifiers import RefusalClassifier

refusal_classifier = RefusalClassifier()

You'll need to create a test case to run deepeval's evals.

Test Run

Running an LLM evaluation creates a test run — a collection of test cases that benchmarks your LLM application at a specific point in time. If you're logged in to Confident AI, the test run also appears there.

Test Cases

In deepeval, a test case represents an LLM interaction and allows you to use the evals you have defined to unit test LLM applications. Single-turn LLM apps are evaluated with an LLMTestCase, while multi-turn LLM apps, such as chatbots, are evaluated with a ConversationalTestCase:

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

answer_relevancy_metric = AnswerRelevancyMetric()
test_case = LLMTestCase(
  input="Who is the current president of the United States of America?",
  actual_output="Joe Biden",
  retrieval_context=["Joe Biden serves as the current president of America."]
)

answer_relevancy_metric.measure(test_case)
print(answer_relevancy_metric.score)

In this example, input mimics an user interaction with a RAG-based LLM application, where actual_output is the output of your LLM application and retrieval_context is the retrieved nodes in your RAG pipeline.

from deepeval.metrics import ConversationCompletenessMetric
from deepeval.test_case import ConversationalTestCase, Turn

conversation_completeness_metric = ConversationCompletenessMetric()
test_case = ConversationalTestCase(
  scenario="User wants to cancel their subscription.",
  turns=[
    Turn(role="user", content="I want to cancel my plan."),
    Turn(role="assistant", content="Done. Your plan is cancelled effective today."),
  ]
)

conversation_completeness_metric.measure(test_case)
print(conversation_completeness_metric.score)

In this example, turns is the conversation between a user and your chatbot, and scenario describes what the user is trying to achieve.

A test case passes only if every metric that carries a verdict succeeds. Two flags let you control which verdicts count:

Metrics with threshold=None still compute scores and reasons, but have no pass/fail opinion and never decide a test case's status. Because every test case must end up passing or failing, each evaluation requires at least one non-flaky metric with a threshold.
Flaky test cases and metrics (flaky=True) have their results recorded and reported as normal, but their failures don't block anything: a flaky metric's failure never fails its test case, and a failing flaky test case makes assert_test() print a warning instead of raising.

Datasets

Datasets in deepeval is a collection of goldens. It provides a centralized interface for you to evaluate a collection of test cases using one or multiple metrics.

from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import AnswerRelevancyMetric
from deepeval import evaluate

answer_relevancy_metric = AnswerRelevancyMetric()
dataset = EvaluationDataset(goldens=[Golden(input="Who is the current president of the United States of America?")])

for golden in dataset.goldens:
  dataset.add_test_case(
    LLMTestCase(
      input=golden.input,
      actual_output=you_llm_app(golden.input)
    )
  )

evaluate(test_cases=dataset.test_cases, metrics=[answer_relevancy_metric])

Goldens

A golden is a test case that hasn't been run through your LLM application yet. It holds the input and the expected results, and becomes a test case once your LLM app produces an output for it. Single-turn LLM apps use a Golden, while multi-turn LLM apps use a ConversationalGolden, which the simulator turns into a full conversation:

from deepeval.dataset import Golden

golden = Golden(
  input="Who is the current president of the United States of America?",
  expected_output="Joe Biden"
)

from deepeval.dataset import ConversationalGolden

golden = ConversationalGolden(
  scenario="User wants to cancel their subscription.",
  expected_outcome="The plan is cancelled and the user receives a confirmation."
)

Synthetic Data

deepeval can generate the data you evaluate with, so you don't have to write every golden or conversation by hand.

Synthesizer

In deepeval, the Synthesizer allows you to generate synthetic datasets. This is especially helpful if you don't have production data or you don't have a golden dataset to evaluate with.

from deepeval.synthesizer import Synthesizer
from deepeval.dataset import EvaluationDataset

synthesizer = Synthesizer()
goldens = synthesizer.generate_goldens_from_docs(
  document_paths=['example.txt', 'example.docx', 'example.pdf']
)

dataset = EvaluationDataset(goldens=goldens)

Simulator

The simulator is only relevant if you're evaluating a multi-turn LLM app, such as a chatbot. It takes your ConversationalGoldens, role-plays a user in each scenario, and records your chatbot's responses turn by turn, producing ConversationalTestCases for you to evaluate.

from deepeval.dataset import ConversationalGolden
from deepeval.simulator import ConversationSimulator
from deepeval.test_case import Turn

async def model_callback(input: str) -> Turn:
  return Turn(role="assistant", content=your_chatbot(input))

simulator = ConversationSimulator(model_callback=model_callback)
test_cases = simulator.simulate(
  conversational_goldens=[
    ConversationalGolden(
      scenario="User wants to cancel their subscription.",
      expected_outcome="The plan is cancelled and the user receives a confirmation."
    )
  ]
)

Unit Testing in CI/CD

deepeval allows you to run evaluations as if you're using Pytest via our Pytest integration. Simply create a test file:

test_example.py

from deepeval import assert_test
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

dataset = EvaluationDataset(goldens=[...])

for golden in dataset.goldens:
  dataset.add_test_case(...) # convert golden to test case

@pytest.mark.parametrize(
    "test_case",
    dataset.test_cases,
)
def test_customer_chatbot(test_case: LLMTestCase):
    assert_test(test_case, [AnswerRelevancyMetric()])

And run the test file in the CLI using deepeval test run:

deepeval test run test_example.py

There are TWO mandatory and ONE optional parameter when calling the assert_test() function:

test_case: an LLMTestCase
metrics: a list of metrics of type BaseMetric
[Optional] run_async: a boolean which when set to True, enables concurrent evaluation of all metrics. Defaulted to True.

You can find the full documentation on running evals in CI/CD, for end-to-end, trajectory-based, and component-level evaluation by clicking on their respective links.

You can include the command as a step in a .yaml file in your CI/CD workflows to run pre-deployment checks on your LLM application.

Evaluating In Scripts

Alternately, you can use deepeval's evaluate function. This approach avoids the CLI (if you're in a notebook environment), and allows for parallel test execution as well.

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset(goldens=[...])
for golden in dataset.goldens:
  dataset.add_test_case(...) # convert golden to test case

evaluate(dataset, [AnswerRelevancyMetric()])

There are TWO mandatory and SIX optional parameters when calling the evaluate() function:

test_cases: a list of LLMTestCases OR ConversationalTestCases, or an EvaluationDataset. You cannot evaluate LLMTestCases and ConversationalTestCases in the same test run.
metrics: a list of metrics of type BaseMetric.
[Optional] hyperparameters: a dict of type dict[str, Union[str, int, float]]. You can log any arbitrary hyperparameter associated with this test run, and it's attached to the run on Confident AI if you're logged in.
[Optional] identifier: a string that allows you to better identify your test run on Confident AI.
[Optional] async_config: an instance of type AsyncConfig that allows you to customize the degree concurrency during evaluation. Defaulted to the default AsyncConfig values.
[Optional] display_config
instance of type DisplayConfig that allows you to customize what is displayed to the console during evaluation. Defaulted to the default DisplayConfig values.
[Optional] error_config: an instance of type ErrorConfig that allows you to customize how to handle errors during evaluation. Defaulted to the default ErrorConfig values.
[Optional] cache_config: an instance of type CacheConfig that allows you to customize the caching behavior during evaluation. Defaulted to the default CacheConfig values.

You can find the full documentation on evaluate(), for both end-to-end and component-level evaluation by clicking on their respective links.

Evaluating Nested Components

You can also run metrics on nested components by setting up tracing in deepeval, and requires under 10 lines of code:

from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import observe, update_current_span
from openai import OpenAI

client = OpenAI()

@observe(metrics=[AnswerRelevancyMetric()])
def complete(query: str):
  response = client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": query}]).choices[0].message.content

  update_current_span(
    test_case=LLMTestCase(input=query, actual_output=response)
  )
  return response

This is very useful especially if you:

Want to run a different set of metrics on different components
Wish to evaluate multiple components at once
Don't want to rewrite your codebase just to bubble up returned variables to create an LLMTestCase

By default, deepeval will not run any metrics when you're running your LLM application outside of evaluate() or assert_test(). For the full guide on evaluating with tracing, visit this page.

Framework Integrations

Framework integrations capture your application's traces and spans so deepeval metrics can evaluate complete agent trajectories or individual components without you recreating the execution structure manually.

Here are a few examples available for your selected SDK:

LangChain

Evaluate complete agent runs, model calls, tools, and retrievers.

Pydantic AI

Evaluate Pydantic AI agents and their instrumented spans.

AWS AgentCore

Evaluate AgentCore-hosted agents and their nested spans.

Google ADK

Evaluate ADK agent runs, tools, and model calls.

Browse all framework integrations →

FAQs

What are the three components an LLM evaluation needs?

A test case (the input and actual_output to score), one or more metrics, and optionally an evaluation dataset to run those metrics across many test cases at once.

When should I run end-to-end vs component-level evals in deepeval?

Use end-to-end when you only care about your system's final input and output. Use component-level (via observe) when you need to score individual retrievers, tools, or agents to see exactly where things break.

Do I need a deepeval account or API key to run evals?

No. deepeval runs entirely locally. The only key you typically need is an LLM provider key such as OPENAI_API_KEY for metrics that use an LLM judge. A Confident AI account is optional and only needed to push results to the cloud.

Should I run evals in CI/CD or with the evaluate() function?

Use deepeval test run when you want evals gated in CI/CD, and evaluate() for scripts where you want parallel execution without the CLI. Both run the same metrics on the same test cases.

How does deepeval score outputs without an exact match to the expected output?

Most of deepeval's 30+ metrics use an LLM judge with techniques like G-Eval, DAG, and QAG rather than string matching, so a correct answer phrased differently from the expected_output can still pass.

Can I find what to evaluate next from production?

Yes. Once your app is live, Confident AI surfaces failing production traces before anyone has written an eval for them and turns them into goldens for your next test run. It's optional; local runs work exactly the same with or without it.

LangChain

Pydantic AI

AWS AgentCore

Google ADK

On this page