LLM Arena Evaluation Quickstart

Learn how to evaluate different versions of your LLM app using LLM Arena-as-a-Judge in deepeval, a comparison-based LLM eval.

Overview

Instead of comparing LLM outputs using a single-output LLM-as-a-Judge method as seen in previous sections, you can also compare n-pairwise test cases to find the best version of your LLM app. This method although does not provide numerical scores, allows you to more reliably choose the "winning" LLM output for a given set of inputs and outputs.

In this 5 min quickstart, you'll learn how to:

Setup an LLM arena
Use Arena G-Eval to pick the best performing LLM app

Prerequisites

Install deepeval
A Confident AI API key (recommended). Sign up for one here

Setup LLM Arena

In deepeval, arena test cases are used to compare different versions of your LLM app to see which one performs better. Each test case is an arena containing different contestants as different versions of your LLM app which are evaluated based on their corresponding LLMTestCase

ArenaGEval picks its winner with an LLM judge, which defaults to OpenAI. You can judge with any model deepeval supports instead:

To use OpenAI for deepeval's LLM metrics, supply your OPENAI_API_KEY in the CLI:

export OPENAI_API_KEY=<your-openai-api-key>

Alternatively, if you're working in a notebook environment (Jupyter or Colab), set your OPENAI_API_KEY in a cell:

%env OPENAI_API_KEY=<your-openai-api-key>

deepeval also allows you to use Azure OpenAI for metrics that are evaluated using an LLM. Run the following command in the CLI to configure your deepeval environment to use Azure OpenAI for all LLM-based metrics.

deepeval set-azure-openai \
    --base-url=<endpoint> \ # e.g. https://example-resource.azure.openai.com/
    --model=<model_name> \ # e.g. gpt-4.1
    --deployment-name=<deployment_name> \  # e.g. Test Deployment
    --api-version=<api_version> \ # e.g. 2025-01-01-preview
    --model-version=<model_version> # e.g. 2024-11-20

Note that the model-version is optional. If you ever wish to stop using Azure OpenAI and move back to regular OpenAI, simply run:

deepeval unset-azure-openai

To use Ollama models for your metrics, run deepeval set-ollama --model=<model> in your CLI. For example:

deepeval set-ollama --model=deepseek-r1:1.5b

Optionally, you can specify the base URL of your local Ollama model instance if you've defined a custom port. The default base URL is set to http://localhost:11434.

deepeval set-ollama --model=deepseek-r1:1.5b \
    --base-url="http://localhost:11434"

To stop using your local Ollama model and move back to OpenAI, run:

deepeval unset-ollama

To use Gemini models with deepeval, run the following command in your CLI.

deepeval set-gemini \
    --model=<model_name> # e.g. "gemini-2.0-flash-001"

deepeval allows you to use ANY custom LLM for evaluation. This includes LLMs from langchain's chat model integrations, Hugging Face's transformers library, or even LLMs in GGML format.

This includes any of your favorite models such as:

Azure OpenAI
Claude via AWS Bedrock
Google Vertex AI
Mistral 7B

All the examples can be found here, but down below is a quick example of a custom Azure OpenAI model through langchain's AzureChatOpenAI module for evaluation:

from langchain_openai import AzureChatOpenAI
from deepeval.models.base_model import DeepEvalBaseLLM

class AzureOpenAI(DeepEvalBaseLLM):
    def __init__(
        self,
        model
    ):
        self.model = model

    def load_model(self):
        return self.model

    def generate(self, prompt: str) -> str:
        chat_model = self.load_model()
        return chat_model.invoke(prompt).content

    async def a_generate(self, prompt: str) -> str:
        chat_model = self.load_model()
        res = await chat_model.ainvoke(prompt)
        return res.content

    def get_model_name(self):
        return "Custom Azure OpenAI Model"

# Replace these with real values
custom_model = AzureChatOpenAI(
    openai_api_version=api_version,
    azure_deployment=azure_deployment,
    azure_endpoint=azure_endpoint,
    openai_api_key=openai_api_key,
)
azure_openai = AzureOpenAI(model=custom_model)
print(azure_openai.generate("Write me a joke"))

When creating a custom LLM evaluation model you should ALWAYS:

inherit DeepEvalBaseLLM.
implement the get_model_name() method, which simply returns a string representing your custom model name.
implement the load_model() method, which will be responsible for returning a model object.
implement the generate() method with one and only one parameter of type string that acts as the prompt to your custom LLM.
the generate() method should return the final output string of your custom LLM. Note that we called chat_model.invoke(prompt).content to access the model generations in this particular example, but this could be different depending on the implementation of your custom model object.
implement the a_generate() method, with the same function signature as generate(). Note that this is an async method. In this example, we called await chat_model.ainvoke(prompt), which is an asynchronous wrapper provided by LangChain's chat models.

Lastly, to use it for evaluation for an LLM-Eval:

from deepeval.metrics import AnswerRelevancyMetric
...

metric = AnswerRelevancyMetric(model=azure_openai)

We CANNOT guarantee that evaluations will work as expected when using a custom model. This is because evaluation requires high levels of reasoning and the ability to follow instructions such as outputting responses in valid JSON formats. To better enable custom LLMs output valid JSONs, read this guide.

Alternatively, if you find yourself running into JSON errors and would like to ignore it, use the -c and -i flag during test run:

deepeval test run test_example.py -i -c

The -i flag ignores errors while the -c flag utilizes the local deepeval cache, so for a partially successful test run you don't have to rerun test cases that didn't error.

Create an arena test case

Create an ArenaTestCase by passing a list of contestants.

main.py

from deepeval.test_case import ArenaTestCase, LLMTestCase, Contestant

contestant_1 = Contestant(
    name="Version 1",
    hyperparameters={"model": "gpt-3.5-turbo"},
    test_case=LLMTestCase(
        input="What is the capital of France?",
        actual_output="Paris",
    ),
)

contestant_2 = Contestant(
    name="Version 2",
    hyperparameters={"model": "gpt-4o"},
    test_case=LLMTestCase(
        input="What is the capital of France?",
        actual_output="Paris is the capital of France.",
    ),
)

contestant_3 = Contestant(
    name="Version 3",
    hyperparameters={"model": "gpt-4.1"},
    test_case=LLMTestCase(
        input="What is the capital of France?",
        actual_output="Absolutely! The capital of France is Paris 😊",
    ),
)

test_case = ArenaTestCase(contestants=[contestant_1, contestant_2, contestant_3])

You can learn more about an ArenaTestCase here.

Define arena metric

The ArenaGEval metric is the only metric that is compatible with ArenaTestCase. It picks a winner among the contestants based on the criteria defined.

from deepeval.metrics import ArenaGEval
from deepeval.test_case import SingleTurnParams

arena_geval = ArenaGEval(
    name="Friendly",
    criteria="Choose the winner of the more friendly contestant based on the input and actual output",
    evaluation_params=[
        SingleTurnParams.INPUT,
        SingleTurnParams.ACTUAL_OUTPUT,
    ]
)

Run Your First Arena Evals

Now that you have created an arena with contestants and defined a metric, you can begin running arena evals to determine the winning contestant.

Run an evaluation

You can run arena evals by using the compare() function.

main.py

from deepeval.test_case import ArenaTestCase, LLMTestCase, SingleTurnParams
from deepeval.metrics import ArenaGEval
from deepeval import compare

test_case = ArenaTestCase(
    contestants=[...], # Use the same contestants you've created before
)

arena_geval = ArenaGEval(...) # Use the same metric you've created before

compare(test_cases=[test_case], metric=arena_geval)

Log prompts and models

Each contestant carries its own hyperparameters, which is how you attribute a winning contestant to the prompt and model that produced it.

from deepeval.prompt import Prompt, PromptMessage
from deepeval.test_case import Contestant, LLMTestCase

prompt_1 = Prompt(
    alias="First Prompt",
    messages_template=[PromptMessage(role="system", content="You are a helpful assistant.")]
)
prompt_2 = Prompt(
    alias="Second Prompt",
    messages_template=[PromptMessage(role="system", content="You are a helpful assistant.")]
)

contestant_1 = Contestant(
    name="Version 1",
    hyperparameters={"model": "gpt-3.5-turbo", "prompt": prompt_1},
    test_case=LLMTestCase(...),
)
contestant_2 = Contestant(
    name="Version 2",
    hyperparameters={"model": "gpt-4o", "prompt": prompt_2},
    test_case=LLMTestCase(...),
)

You can now run this file to get your results:

bash

python main.py

This should let you see the results of the arena as shown below:

Counter({'Version 3': 1})

🎉🥳 Congratulations! You have just ran your first LLM arena-based evaluation. Here's what happened:

When you call compare(), deepeval loops through each ArenaTestCase
For each test case, deepeval uses the ArenaGEval metric to pick the "winner"
To make the arena unbiased, deepeval masks the names of each contestant and randomizes their positions
In the end, you get the number of "wins" each contestant got as the final output.

Unlike single-output LLM-as-a-Judge (which is everything but LLM arena evals), the concept of a "passing" test case does not exist for arena evals.

View on Confident AI (recommended)

If you've set your CONFIDENT_API_KEY, your arena comparisons will automatically appear as an experiment on Confident AI, which deepeval integrates with natively.

Next Steps

deepeval lets you run Arena comparisons locally but isn’t optimized for iterative prompt or model improvements. If you’re looking for a more comprehensive and streamlined way to run Arena comparisons, Confident AI enables you to easily test different prompts, models, tools, and output configurations side by side, and evaluate them using any deepeval metric beyond ArenaGEval—all directly on the platform.

Compare model outputs directly using arena evaluations.

Create an experiment to run comprehensive comparisons on an evaluation dataset and set of metrics.

View detailed traces of LLM and tool calls during model comparisons.

Apply custom evaluation metrics to determine winning models in head-to-head comparisons.

Track prompts and model configurations to understand which hyperparameters lead to better performance.

Now that you have run your first Arena evals, you should:

Customize your metrics: You can change the criteria of your metric to be more specific to your use-case.
Prepare a dataset: If you don't have one, generate one as a starting point to store your inputs as goldens.

The arena metric is only used for picking winners among the contestants, it's not used for evaluating the answers themselves. To evaluate your LLM application on specific use cases you can read the other quickstarts here: