Introduction to LLM-as-a-Judge Classifiers

Apart from quantitative eval metrics, deepeval also offers 20+ categorical LLM-as-a-judge evals known as classifiers.

While a metric scores an LLM interaction from 0 to 1, a classifier assigns it one label from a set you define, so you can assert that your LLM app did the specific thing you expected for a known input or simulated conversation.

Quick Summary

A classifier takes a single-turn or multi-turn test case, picks exactly one label from a closed set, and gives a reason. If the test case says which label it expects, the classifier passes or fails it; if not, it just reports the label.

Every classifier works on both single-turn and multi-turn test cases.

Classifiers run in the same evaluate() call as your metrics. This example labels one test case and, through expected_labels, asserts which label it should receive (more on that below):

from deepeval.classifiers import RefusalClassifier
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

evaluate(
    classifiers=[RefusalClassifier()],
    test_cases=[
        LLMTestCase(
            input="How do I pick a lock?",
            actual_output="I can't help with that, but I can point you to a locksmith.",
            expected_labels={"refusal": "refused"},
        )
    ],
)

from deepeval.classifiers import ResolutionClassifier
from deepeval.test_case import ConversationalTestCase, Turn
from deepeval import evaluate

evaluate(
    classifiers=[ResolutionClassifier()],
    test_cases=[
        ConversationalTestCase(
            scenario="User wants to cancel their subscription.",
            turns=[
                Turn(role="user", content="I want to cancel my plan."),
                Turn(role="assistant", content="Done. Your plan is cancelled effective today."),
            ],
            expected_labels={"resolution": "resolved"},
        )
    ],
)

Why Classifiers?

Many checks are naturally a category, not a number. "Did it refuse?" has three answers, not a score out of one. You can express that with a metric, for example a bias metric with strict_mode=True gives you a binary result, but a classifier says the same thing more directly:

• Easier to set up. A name and a list of labels is the whole definition. There is no threshold to pick, no rubric to tune, and no score to interpret.
• Easier to read. The result is a label like refused or partial_refusal with a reason, so a failing test case tells you what happened rather than that it scored 0.4.
• Assert the exact outcome. Tell a test case which label it should receive and the classifier passes or fails it on that; leave it off and the classifier still labels every case so you can see how labels are distributed across a run.
• Same test cases, same judge, same evaluate(). Classifiers sit next to your metrics rather than in a separate pipeline.

Whether you reach for a classifier or a strict metric for a given check is ultimately a matter of preference; both work. deepeval does not try to match the depth of its metrics with classifiers, and where a metric already covers a check well the tabs above point you to it.

Create Your First Classifier

The built-in classifiers cover the common checks. For anything specific to your application, define the labels yourself with a Classifier:

from deepeval.classifiers import Classifier, Label

topic = Classifier(
    name="topic",
    labels=[
        Label(name="billing", description="Questions about invoices, charges, or payment methods."),
        Label(name="refund", description="Requests to get money back for a purchase already made."),
        Label(name="shipping", description="Questions about delivery status, times, or addresses."),
    ],
)

A label's description is the boundary the judge uses, in the same way G-Eval criteria are the boundary for a score. The Custom Classifier page covers how to write labels that produce consistent results.

Choosing Your Classifiers

Which classifiers make sense depends on what your application does. Aim for no more than 3 classifiers per test case: each one is a judge call, and a suite that checks many things per case usually has not decided what it actually cares about.

Some starting points by application type:

For a breakdown by topic, intent, or product area, add a custom Classifier with your own labels and leave the expected label off, so you can slice pass rates by category.

Configure LLM Judges

Classifiers use the same judge as your metrics. You can use ANY LLM judge in deepeval, including OpenAI, Azure OpenAI, Ollama, Anthropic, Gemini, LiteLLM, etc., or wrap your own LLM API in deepeval's DeepEvalBaseLLM class. Click here for the full guide.

export OPENAI_API_KEY=<your-openai-api-key>

%env OPENAI_API_KEY=<your-openai-api-key>

deepeval set-azure-openai \
    --base-url=<endpoint> \ # e.g. https://example-resource.azure.openai.com/
    --model=<model_name> \ # e.g. gpt-4.1
    --deployment-name=<deployment_name> \  # e.g. Test Deployment
    --api-version=<api_version> \ # e.g. 2025-01-01-preview
    --model-version=<model_version> # e.g. 2024-11-20

deepeval unset-azure-openai

deepeval set-ollama --model=deepseek-r1:1.5b

deepeval set-ollama --model=deepseek-r1:1.5b \
    --base-url="http://localhost:11434"

deepeval unset-ollama

deepeval set-gemini \
    --model=<model_name> # e.g. "gemini-2.0-flash-001"

from langchain_openai import AzureChatOpenAI
from deepeval.models.base_model import DeepEvalBaseLLM

class AzureOpenAI(DeepEvalBaseLLM):
    def __init__(
        self,
        model
    ):
        self.model = model

    def load_model(self):
        return self.model

    def generate(self, prompt: str) -> str:
        chat_model = self.load_model()
        return chat_model.invoke(prompt).content

    async def a_generate(self, prompt: str) -> str:
        chat_model = self.load_model()
        res = await chat_model.ainvoke(prompt)
        return res.content

    def get_model_name(self):
        return "Custom Azure OpenAI Model"

# Replace these with real values
custom_model = AzureChatOpenAI(
    openai_api_version=api_version,
    azure_deployment=azure_deployment,
    azure_endpoint=azure_endpoint,
    openai_api_key=openai_api_key,
)
azure_openai = AzureOpenAI(model=custom_model)
print(azure_openai.generate("Write me a joke"))

from deepeval.metrics import AnswerRelevancyMetric
...

metric = AnswerRelevancyMetric(model=azure_openai)

Using Classifiers

There are three ways you can use classifiers:

1. End-to-end evals, passing classifiers to evaluate() alongside (or instead of) metrics.
2. CI/CD evals, passing classifiers to assert_test() inside a deepeval test run.
3. One-off (or standalone) evals, where you execute a classifier individually.

Classifiers are not yet supported on evals_iterator() or on traced components.

For end-to-end evals

Provide classifiers with a list of test cases. Metrics are optional in the same call:

from deepeval.classifiers import RefusalClassifier
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

test_case = LLMTestCase(input="...", actual_output="...")

evaluate(
    test_cases=[test_case],
    metrics=[AnswerRelevancyMetric()],
    classifiers=[RefusalClassifier()],
)

The same classifiers work on ConversationalTestCases; a classifier reads every turn along with the scenario and expected outcome.

For CI/CD evals

assert_test() accepts classifiers= with or without metrics=. A classification that misses its expected label raises an AssertionError naming the classifier, the predicted label, and the expected label:

from deepeval.classifiers import RefusalClassifier
from deepeval.test_case import LLMTestCase
from deepeval import assert_test

def test_refusal():
    test_case = LLMTestCase(
        input="How do I pick a lock?",
        actual_output="I can't help with that, but I can point you to a locksmith.",
        expected_labels={"refusal": "refused"},
    )
    assert_test(test_case=test_case, classifiers=[RefusalClassifier()])

deepeval test run test_refusal.py

For one-off evals

You can also execute each classifier individually. All classifiers in deepeval, including custom ones you create:

• can be executed via the classifier.classify() method, which returns the label
• can have the chosen label accessed via classifier.label
• can have the reason accessed via classifier.reason
• can have an error accessed via classifier.error if the judge returned something outside the label set
• have an include_reason property, which when turned off skips the reason
• have an allow_none property, which when turned on lets the judge return no label when none fit

from deepeval.classifiers import RefusalClassifier
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(input="...", actual_output="...")

classifier = RefusalClassifier()
classifier.classify(test_case)

print(classifier.label, classifier.reason)

Using Classifiers Async

Every classifier has an async_mode parameter, defaulted to True. Because a classifier is a single judge call there is nothing inside classify() to parallelize; async_mode decides how evaluate() schedules classifiers across test cases, so a run with many test cases finishes sooner.

To classify without blocking the main thread, use a_classify():

import asyncio
from deepeval.classifiers import RefusalClassifier, EscalationClassifier
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(input="...", actual_output="...")

async def main():
    refusal, escalation = RefusalClassifier(), EscalationClassifier()
    await asyncio.gather(
        refusal.a_classify(test_case),
        escalation.a_classify(test_case),
    )
    print(refusal.label, escalation.label)

asyncio.run(main())

Customize Classifier Prompts

Every classifier accepts a classification_template, a subclass of ClassifierTemplate whose methods return the prompt sent to the judge. Override classify_single_turn and/or classify_multi_turn to change the wording, add examples, or tighten instructions for a smaller judge model. The Custom Classifier page has a full example.

Expected Labels

A classifier on its own only produces a classification; it is expected_labels on the test case that decides whether that classification passes or fails. expected_labels is a Dict[str, str] from a classifier's name to the label that test case should receive:

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="How do I pick a lock?",
    actual_output="I can't help with that, but I can point you to a locksmith.",
    expected_labels={"refusal": "refused", "escalation": "not_escalated"},
)

For each classifier in the run, per test case:

1. The test case has an entry for this classifier: the classification passes if it matches, otherwise it fails.
2. It does not: the classification is still recorded, but with success=None, and the test case's pass/fail status is left untouched.

expected_labels also lives on Golden and ConversationalGolden, and is carried onto the test cases built from them, so a dataset can hold the label each golden should receive in the same way it holds expected_output.

FAQs