Introduction to LLM Evals

Q: What are the three components an LLM evaluation needs?

A test case (the input and actual_output to score), one or more metrics, and optionally an evaluation dataset to run those metrics across many test cases at once.

Q: Do I need a deepeval account or API key to run evals?

No. deepeval runs entirely locally. The only key you typically need is an LLM provider key such as OPENAI_API_KEY for metrics that use an LLM judge. A Confident AI account is optional and only needed to push results to the cloud.

Q: Should I run evals with deepeval test run or the evaluate() function?

Use deepeval test run with pytest when you want evals gated in CI/CD, and evaluate() for scripts or notebooks where you want parallel execution without the CLI. Both run the same metrics on the same test cases.

Q: How does deepeval score outputs without an exact match to expected_output?

Most of deepeval's 30+ metrics use an LLM judge with techniques like G-Eval, DAG, and QAG rather than string matching, so a correct answer phrased differently from the expected_output can still pass.

Quick Summary

Evaluation refers to the process of testing your LLM application outputs, and requires the following components:

Test cases
Metrics
Evaluation dataset

Here's a diagram of what an ideal evaluation workflow looks like using deepeval:

There are TWO types of LLM evaluations in deepeval:

End-to-end evaluation: The overall input and outputs of your LLM system.
Component-level evaluation: The individual inner workings of your LLM system.

Both can be done using either deepeval test run in CI/CD pipelines, or via the evaluate() function in Python scripts.

Test Run

Running an LLM evaluation creates a test run — a collection of test cases that benchmarks your LLM application at a specific point in time. If you're logged into Confident AI, you'll also receive a fully sharable LLM testing report on the cloud.

Metrics

deepeval offers 30+ evaluation metrics, most of which are evaluated using LLMs (visit the metrics section to learn why).

from deepeval.metrics import AnswerRelevancyMetric

answer_relevancy_metric = AnswerRelevancyMetric()

You'll need to create a test case to run deepeval's metrics.

Test Cases

In deepeval, a test case represents an LLM interaction and allows you to use evaluation metrics you have defined to unit test LLM applications.

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
  input="Who is the current president of the United States of America?",
  actual_output="Joe Biden",
  retrieval_context=["Joe Biden serves as the current president of America."]
)

In this example, input mimics an user interaction with a RAG-based LLM application, where actual_output is the output of your LLM application and retrieval_context is the retrieved nodes in your RAG pipeline. Creating a test case allows you to evaluate using deepeval's default metrics:

from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

answer_relevancy_metric = AnswerRelevancyMetric()
test_case = LLMTestCase(
  input="Who is the current president of the United States of America?",
  actual_output="Joe Biden",
  retrieval_context=["Joe Biden serves as the current president of America."]
)

answer_relevancy_metric.measure(test_case)
print(answer_relevancy_metric.score)

A test case passes only if every metric that carries a verdict succeeds. Two flags let you control which verdicts count:

Metrics with threshold=None still compute scores and reasons, but have no pass/fail opinion and never decide a test case's status. Because every test case must end up passing or failing, each evaluation requires at least one non-flaky metric with a threshold.
Flaky test cases and metrics (flaky=True) have their results recorded and reported as normal, but their failures don't block anything: a flaky metric's failure never fails its test case, and a failing flaky test case makes assert_test() print a warning instead of raising.

Datasets

Datasets in deepeval is a collection of goldens. It provides a centralized interface for you to evaluate a collection of test cases using one or multiple metrics.

from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import AnswerRelevancyMetric
from deepeval import evaluate

answer_relevancy_metric = AnswerRelevancyMetric()
dataset = EvaluationDataset(goldens=[Golden(input="Who is the current president of the United States of America?")])

for golden in dataset.goldens:
  dataset.add_test_case(
    LLMTestCase(
      input=golden.input,
      actual_output=you_llm_app(golden.input)
    )
  )

evaluate(test_cases=dataset.test_cases, metrics=[answer_relevancy_metric])

Synthesizer

In deepeval, the Synthesizer allows you to generate synthetic datasets. This is especially helpful if you don't have production data or you don't have a golden dataset to evaluate with.

from deepeval.synthesizer import Synthesizer
from deepeval.dataset import EvaluationDataset

synthesizer = Synthesizer()
goldens = synthesizer.generate_goldens_from_docs(
  document_paths=['example.txt', 'example.docx', 'example.pdf']
)

dataset = EvaluationDataset(goldens=goldens)

Evaluating With Pytest

deepeval allows you to run evaluations as if you're using Pytest via our Pytest integration. Simply create a test file:

test_example.py

from deepeval import assert_test
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

dataset = EvaluationDataset(goldens=[...])

for golden in dataset.goldens:
  dataset.add_test_case(...) # convert golden to test case

@pytest.mark.parametrize(
    "test_case",
    dataset.test_cases,
)
def test_customer_chatbot(test_case: LLMTestCase):
    assert_test(test_case, [AnswerRelevancyMetric()])

And run the test file in the CLI using deepeval test run:

deepeval test run test_example.py

There are TWO mandatory and ONE optional parameter when calling the assert_test() function:

test_case: an LLMTestCase
metrics: a list of metrics of type BaseMetric
[Optional] run_async: a boolean which when set to True, enables concurrent evaluation of all metrics. Defaulted to True.

You can find the full documentation on deepeval test run, for both end-to-end and component-level evaluation by clicking on their respective links.

You can include the deepeval test run command as a step in a .yaml file in your CI/CD workflows to run pre-deployment checks on your LLM application.

Evaluating Without Pytest

Alternately, you can use deepeval's evaluate function. This approach avoids the CLI (if you're in a notebook environment), and allows for parallel test execution as well.

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset(goldens=[...])
for golden in dataset.goldens:
  dataset.add_test_case(...) # convert golden to test case

evaluate(dataset, [AnswerRelevancyMetric()])

There are TWO mandatory and SIX optional parameters when calling the evaluate() function:

test_cases: a list of LLMTestCases OR ConversationalTestCases, or an EvaluationDataset. You cannot evaluate LLMTestCases and ConversationalTestCases in the same test run.
metrics: a list of metrics of type BaseMetric.
[Optional] hyperparameters: a dict of type dict[str, Union[str, int, float]]. You can log any arbitrary hyperparameter associated with this test run to pick the best hyperparameters for your LLM application on Confident AI.
[Optional] identifier: a string that allows you to better identify your test run on Confident AI.
[Optional] async_config: an instance of type AsyncConfig that allows you to customize the degree concurrency during evaluation. Defaulted to the default AsyncConfig values.
[Optional] display_config
instance of type DisplayConfig that allows you to customize what is displayed to the console during evaluation. Defaulted to the default DisplayConfig values.
[Optional] error_config: an instance of type ErrorConfig that allows you to customize how to handle errors during evaluation. Defaulted to the default ErrorConfig values.
[Optional] cache_config: an instance of type CacheConfig that allows you to customize the caching behavior during evaluation. Defaulted to the default CacheConfig values.

You can find the full documentation on evaluate(), for both end-to-end and component-level evaluation by clicking on their respective links.

Evaluating Nested Components

You can also run metrics on nested components by setting up tracing in deepeval, and requires under 10 lines of code:

from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import observe, update_current_span
from openai import OpenAI

client = OpenAI()

@observe(metrics=[AnswerRelevancyMetric()])
def complete(query: str):
  response = client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": query}]).choices[0].message.content

  update_current_span(
    test_case=LLMTestCase(input=query, output=response)
  )
  return response

This is very useful especially if you:

Want to run a different set of metrics on different components
Wish to evaluate multiple components at once
Don't want to rewrite your codebase just to bubble up returned variables to create an LLMTestCase

By default, deepeval will not run any metrics when you're running your LLM application outside of evaluate() or assert_test(). For the full guide on evaluating with tracing, visit this page.

FAQs

What are the three components an LLM evaluation needs?

A test case (the input and actual_output to score), one or more metrics, and optionally an evaluation dataset to run those metrics across many test cases at once.

When should I run end-to-end vs component-level evals in deepeval?

Use end-to-end when you only care about your system's final input and output. Use component-level (via the @observe decorator) when you need to score individual retrievers, tools, or agents to see exactly where things break.

Do I need a deepeval account or API key to run evals?

No. deepeval runs entirely locally. The only key you typically need is an LLM provider key such as OPENAI_API_KEY for metrics that use an LLM judge. A Confident AI account is optional and only needed to push results to the cloud.

Should I run evals with deepeval test run or the evaluate() function?

Use deepeval test run with pytest when you want evals gated in CI/CD, and evaluate() for scripts or notebooks where you want parallel execution without the CLI. Both run the same metrics on the same test cases.

How does deepeval score outputs without an exact match to expected_output?

Most of deepeval's 30+ metrics use an LLM judge with techniques like G-Eval, DAG, and QAG rather than string matching, so a correct answer phrased differently from the expected_output can still pass.

Can my team keep deepeval results on the cloud and review them in a shared UI?

Every test run is local-first, but when you're logged into Confident AI (the platform built by the deepeval team) the same run produces a sharable testing report on the cloud. That gives a team a shared UI to compare runs, track regressions over time, and inspect failing cases — without changing any of your evaluation code. It's optional; local runs work exactly the same with or without it.

On this page