G-Eval

Q: Why are my G-Eval scores inconsistent between runs?

Usually relying on criteria alone, which makes the metric regenerate fresh CoT steps every run. Provide explicit to skip that, and add a rubric or to tighten scoring.

Q: Should I define criteria or evaluation steps?

criteria when exploring and you want G-Eval to draft the steps; for reproducible, controllable scores. Start with a short criteria, inspect the generated steps, then lock them in. Pass only one, never both.

Q: When should I use the multi-turn version instead of single-turn?

Use ConversationalGEval when your criteria spans a whole ConversationalTestCase — tone consistency, memory of earlier context, recovery after a bad turn. Plain GEval sees only one input/ pair and can't reason across turns. Neither class accepts the other's test case type.

Q: Does the multi-turn version score each turn separately or the whole conversation?

The whole conversation. ConversationalGEval considers the entire turns history and returns a single score and reason, not one per turn.

Q: Should I add every test case parameter to evaluation params?

No. Only include parameters referenced in your criteria/. Unused params (e.g. your steps never mention, or MultiTurnParams.TOOLS_CALLED when tool usage isn't part of the criteria) feed the judge irrelevant context and degrade the score.

Q: How do I stop G-Eval from clustering scores in the middle?

Pass a list of Rubrics via the rubric argument to confine the judge to specific score bands (non-overlapping s within 0-10). For pass/fail, set to force a binary 1 or 0.

LLM-as-a-judge

Custom

Single-turn

Multi-turn

Multimodal

G-Eval is a framework that uses LLM-as-a-judge with chain-of-thoughts (CoT) to evaluate LLM outputs based on ANY custom criteria. The G-Eval metric is the most versatile type of metric deepeval has to offer, and is capable of evaluating almost any use case with human-like accuracy.

Usually, a GEval metric will be used alongside one of the other metrics that are more system specific (such as ContextualRelevancyMetric for RAG, and TaskCompletionMetric for agents). This is because G-Eval is a custom metric best for subjective, use case specific evaluation.

Usage

To create a custom metric that uses LLMs for evaluation, simply instantiate a G-Eval metric and define an evaluation criteria in everyday language:

from deepeval.test_case import SingleTurnParams
from deepeval.metrics import GEval

correctness_metric = GEval(
    name="Correctness",
    criteria="Determine whether the actual output is factually correct based on the expected output.",
    # NOTE: you can only provide either criteria or evaluation_steps, and not both
    evaluation_steps=[
        "Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
        "You should also heavily penalize omission of detail",
        "Vague language, or contradicting OPINIONS, are OK"
    ],
    evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
)

from deepeval.test_case import MultiTurnParams
from deepeval.metrics import ConversationalGEval

professionalism_metric = ConversationalGEval(
    name="Professionalism",
    criteria="Determine whether the assistant has acted professionally based on the content.",
    # NOTE: you can only provide either criteria or evaluation_steps, and not both
    evaluation_steps=[
        "Check whether every assistant turn maintains a courteous, professional tone",
        "Heavily penalize slang, sarcasm, or dismissive language towards the user",
        "Hedged or uncertain answers are OK as long as they stay respectful"
    ],
    evaluation_params=[MultiTurnParams.CONTENT],
)

There are THREE mandatory and NINE optional parameters required when instantiating a G-Eval metric:

name: name of custom metric.
criteria: a description outlining the specific evaluation aspects for each test case.
evaluation_params: a list of type SingleTurnParams (or MultiTurnParams for multi-turn). Include only the parameters that are relevant for evaluation. For single-turn G-Eval, omit it to evaluate the trajectory instead.
[Optional] evaluation_steps: a list of strings outlining the exact steps the LLM should take for evaluation. If evaluation_steps is not provided, G-Eval will generate a series of evaluation_steps on your behalf based on the provided criteria.
[Optional] rubric: a list of Rubrics that allows you to confine the range of the final metric score.
[Optional] threshold: the passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to gpt-5.4.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
[Optional] evaluation_template: a class of type GEvalTemplate, which allows you to override the default prompts used to compute the G-Eval score. Defaulted to deepeval's GEvalTemplate.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.

As mentioned in the metrics introduction section, all of deepeval's metrics return a score ranging from 0 - 1, and a metric is only successful if the evaluation score is equal to or greater than threshold, and G-Eval is no exception. You can access the score and reason for each individual metric:

from deepeval.test_case import LLMTestCase
from deepeval import evaluate
...

test_case = LLMTestCase(
    input="The dog chased the cat up the tree, who ran up the tree?",
    actual_output="It depends, some might consider the cat, while others might argue the dog.",
    expected_output="The cat."
)

# To run metric as a standalone
# correctness_metric.measure(test_case)
# print(correctness_metric.score, correctness_metric.reason)

evaluate(test_cases=[test_case], metrics=[correctness_metric])

from deepeval.test_case import ConversationalTestCase, Turn
from deepeval import evaluate
...

convo_test_case = ConversationalTestCase(
    turns=[
        Turn(role="user", content="My order arrived a week late. I want a refund."),
        Turn(role="assistant", content="Sorry about that. What is your order number?"),
        Turn(role="user", content="It's 4471."),
        Turn(role="assistant", content="Thanks, I've issued a full refund. It'll land in 5-7 days."),
    ]
)

# To run metric as a standalone
# professionalism_metric.measure(convo_test_case)
# print(professionalism_metric.score, professionalism_metric.reason)

evaluate(test_cases=[convo_test_case], metrics=[professionalism_metric])

You can upload your G-Eval metrics to Confident AI and use them as custom evaluation metrics. To upload a metric simply call the upload method of a metric instance:

...

metric = GEval(...)
metric.upload()

Evaluation Steps

Providing evaluation_steps tells G-Eval to follow your evaluation_steps for evaluation instead of first generating one from criteria, which allows for more controllable metric scores (more info here):

...

correctness_metric = GEval(
    name="Correctness",
    evaluation_steps=[
        "Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
        "You should also heavily penalize omission of detail",
        "Vague language, or contradicting OPINIONS, are OK"
    ],
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
)

...

professionalism_metric = ConversationalGEval(
    name="Professionalism",
    evaluation_steps=[
        "Check whether every assistant turn maintains a courteous, professional tone",
        "Heavily penalize slang, sarcasm, or dismissive language towards the user",
        "Hedged or uncertain answers are OK as long as they stay respectful"
    ],
    evaluation_params=[MultiTurnParams.CONTENT],
)

Rubric

You can provide a list of Rubrics through the rubric argument to confine your evaluation LLM to output in specific score ranges:

from deepeval.metrics.g_eval import Rubric
...

correctness_metric = GEval(
    name="Correctness",
    criteria="Determine whether the actual output is factually correct based on the expected output.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
    rubric=[
        Rubric(score_range=(0,2), expected_outcome="Factually incorrect."),
        Rubric(score_range=(3,6), expected_outcome="Mostly correct."),
        Rubric(score_range=(7,9), expected_outcome="Correct but missing minor details."),
        Rubric(score_range=(10,10), expected_outcome="100% correct."),
    ]
)

from deepeval.metrics.g_eval import Rubric
...

professionalism_metric = ConversationalGEval(
    name="Professionalism",
    criteria="Determine whether the assistant has acted professionally based on the content.",
    evaluation_params=[MultiTurnParams.CONTENT],
    rubric=[
        Rubric(score_range=(0,2), expected_outcome="Rude or dismissive in at least one turn."),
        Rubric(score_range=(3,6), expected_outcome="Mostly professional with occasional lapses."),
        Rubric(score_range=(7,9), expected_outcome="Professional throughout, minor informality."),
        Rubric(score_range=(10,10), expected_outcome="Flawlessly professional in every turn."),
    ]
)

Note that score_range ranges from 0 - 10, inclusive and different Rubrics must not have overlapping score_ranges. You can also specify score_ranges where the start and end values are the same to represent a single score.

Within components

You can also run GEval within nested components for component-level evaluation. This only works for single-turn G-Eval.

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.tracing import observe, update_current_span
...

@observe(metrics=[correctness_metric])
def inner_component():
    # Component can be anything from an LLM call, retrieval, agent, tool use, etc.
    update_current_span(test_case=LLMTestCase(input="...", actual_output="..."))
    return

@observe
def llm_app(input: str):
    inner_component()
    return

dataset = EvaluationDataset(goldens=[Golden(input="Hi!")])
for golden in dataset.evals_iterator():
    llm_app(golden.input)

Evaluating the trajectory

When you leave out evaluation_params, single-turn GEval judges the whole trace, not individual test case fields. That covers every agent, LLM, tool, and retriever span with its inputs and outputs. Use it for criteria about how your app reached its answer, such as tool choice, the arguments passed, or whether retrieved results were used correctly.

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.tracing import observe
from deepeval.metrics import GEval

trajectory = GEval(
    name="Tool Use",
    criteria="Did the agent call the right tool with the right arguments, and does its final answer match what the tool returned?",
)

@observe(type="tool")
def order_lookup(order_id: str):
    return {"status": "in_transit"}

@observe(type="agent")
def support_agent(input: str):
    return f"Your order is {order_lookup('1234')['status']}."

dataset = EvaluationDataset(goldens=[Golden(input="Where is my order #1234?")])
for golden in dataset.evals_iterator(metrics=[trajectory]):
    support_agent(golden.input)

Passed to evals_iterator, the metric judges the full trace. Attached to a component's metrics within components, it judges only that span and its children.

As a standalone

You can also run G-Eval on a single test case as a standalone, one-off execution.

...

correctness_metric.measure(test_case)
print(correctness_metric.score, correctness_metric.reason)

...

professionalism_metric.measure(convo_test_case)
print(professionalism_metric.score, professionalism_metric.reason)

What is G-Eval?

G-Eval is a framework originally from the paper "NLG Evaluation using GPT-4 with Better Human Alignment" that uses LLMs to evaluate LLM outputs (aka. LLM-Evals), and is one the best ways to create task-specific metrics.

The G-Eval algorithm first generates a series of evaluation steps for chain of thoughts (CoTs) prompting before using the generated steps to determine the final score via a "form-filling paradigm" (which is just a fancy way of saying G-Eval requires different test case parameters for evaluation depending on the generated steps).

After generating a series of evaluation steps, G-Eval will:

Create prompt by concatenating the evaluation steps with all the parameters in an LLMTestCase that is supplied to evaluation_params. For ConversationalGEval, the parameters are pulled from every Turn in the ConversationalTestCase and laid out as the full conversation history.
At the end of the prompt, ask it to generate a score between 1–5, where 5 is better than 1.
Take the probabilities of the output tokens from the LLM to normalize the score and take their weighted summation as the final result.

Here are the results from the paper, which shows how G-Eval outperforms all traditional, non-LLM evals that were mentioned earlier in this article:

How Is It Calculated?

Since G-Eval is a two-step algorithm that generates chain of thoughts (CoTs) for better evaluation, in deepeval this means first generating a series of evaluation_steps using CoT based on the given criteria, before using the generated steps to determine the final score using the parameters presented in the test case.

When you provide evaluation_steps, the metric skips the first step and uses the provided steps to determine the final score instead, make it more reliable across different runs. If you don't have a clear evaluation_stepss, what we've found useful is to first write a criteria which can be extremely short, and use the evaluation_steps generated by G-Eval for subsequent evaluation and fine-tuning of criteria.

The only difference between single-turn and multi-turn G-Eval is what the judge sees: single-turn presents the fields of one test case, while multi-turn presents the entire conversation history. Both return a single score and reason; multi-turn G-Eval does not score turns individually.

In the original G-Eval paper, the authors used the probabilities of the LLM output tokens to normalize the score by calculating a weighted summation.

This step was introduced in the paper because it minimizes bias in LLM scoring. This normalization step is automatically handled by deepeval by default (unless you're using a custom model).

See how to implement your cusotm logprobs

To enable weighted summation for a custom model, implement a_generate_raw_response (and optionally generate_raw_response) on your DeepEvalBaseLLM subclass. The method must return a tuple of (ChatCompletion, cost) where the ChatCompletion object contains logprobs in its response:

from openai.types.chat import ChatCompletion

class MyCustomModel(DeepEvalBaseLLM):
    async def a_generate_raw_response(self, prompt: str, **kwargs) -> tuple:
        # Call your LLM with logprobs enabled (top_logprobs is passed via kwargs)
        response: ChatCompletion = await self.client.chat.completions.create(
            model=self.model_name,
            messages=[{"role": "user", "content": prompt}],
            logprobs=True,
            top_logprobs=kwargs.get("top_logprobs", 5),
        )
        cost = 0  # replace with actual cost if available
        return response, cost

deepeval will automatically use the logprobs in the returned ChatCompletion to compute the weighted summation score. If a_generate_raw_response is not implemented, deepeval falls back to the raw integer score from a_generate.

Examples

deepeval runs more than 10 million G-Eval metrics a day (we wrote a blog about it here), and in this section we will list out the two most common use cases we see. The rest, each in single-turn and multi-turn form, live in the G-Eval examples guide.

Answer Correctness

Answer correctness is the most used G-Eval metric of all and usually involves comparing the actual_output to the expected_output, which makes it a reference-based metric.

from deepeval.test_case import SingleTurnParams
from deepeval.metrics import GEval

correctness = GEval(
    name="Correctness",
    evaluation_steps=[
        "Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
        "You should also heavily penalize omission of detail",
        "Vague language, or contradicting OPINIONS, are OK"
    ],
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
)

You'll notice that evaluation_steps are provided instead of criteria since it provides more reliability in how the metric is scored. For the full example, click here.

Coherence

Coherence is usually a referenceless metric that covers several criteria such as fluency, consistency, and clarify. Below is an example of using GEval to assess clarify in the coherence spectrum of criteria:

from deepeval.test_case import SingleTurnParams
from deepeval.metrics import GEval

clarity = GEval(
    name="Clarity",
    evaluation_steps=[
        "Evaluate whether the response uses clear and direct language.",
        "Check if the explanation avoids jargon or explains it when used.",
        "Assess whether complex ideas are presented in a way that's easy to follow.",
        "Identify any vague or confusing parts that reduce understanding."
    ],
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
)

Full example and advice on best practices available here.

For tonality, safety, and custom RAG metrics, see the G-Eval examples guide.

Customize Your Template

Since deepeval's G-Eval is evaluated by LLM-as-a-judge, you can likely improve your metric accuracy by overriding deepeval's default prompt templates. This is especially helpful if:

You're using a custom evaluation LLM, especially for smaller models that have weaker instruction following capabilities.
You want to customize the examples used in the default GEvalTemplate to better align with your expectations.

Here's a quick example of how you can override the process of generating evaluation steps in the G-Eval algorithm:

from deepeval.metrics.g_eval import GEvalTemplate
from deepeval.metrics import GEval
import textwrap

# Define custom template
class CustomGEvalTemplate(GEvalTemplate):
    @staticmethod
    def generate_evaluation_steps(parameters: str, criteria: str):
        return textwrap.dedent(
            f"""
            You are given evaluation criteria for assessing {parameters}. Based on the criteria,
            produce 3-4 clear steps that explain how to evaluate the quality of {parameters}.

            Criteria:
            {criteria}

            Return JSON only, in this format:
            {{
                "steps": [
                    "Step 1",
                    "Step 2",
                    "Step 3"
                ]
            }}

            JSON:
            """
        )

# Inject custom template to metric
metric = GEval(evaluation_template=CustomGEvalTemplate)
metric.measure(...)

from deepeval.metrics.conversational_g_eval import ConversationalGEvalTemplate
from deepeval.metrics import ConversationalGEval
import textwrap

# Define custom template
class CustomConvoGEvalTemplate(ConversationalGEvalTemplate):
    @staticmethod
    def generate_evaluation_steps(parameters: str, criteria: str):
        return textwrap.dedent(
            f"""
            You are given criteria for evaluating a conversation based on the following parameters: {parameters}.
            Write 3-4 clear and concise evaluation steps that describe how to judge the quality of each turn and the conversation overall.

            Criteria:
            {criteria}

            Return JSON only in the format:
            {{
                "steps": [
                    "Step 1",
                    "Step 2",
                    "Step 3"
                ]
            }}

            JSON:
            """
        )

# Inject custom template to metric
metric = ConversationalGEval(evaluation_template=CustomConvoGEvalTemplate)
metric.measure(...)

FAQs

Why are my G-Eval scores inconsistent between runs?

Usually relying on criteria alone, which makes the metric regenerate fresh CoT steps every run. Provide explicit evaluation_steps to skip that, and add a rubric or strict_mode to tighten scoring.

Should I define criteria or evaluation steps?

criteria when exploring and you want G-Eval to draft the steps; evaluation_steps for reproducible, controllable scores. Start with a short criteria, inspect the generated steps, then lock them in. Pass only one, never both.

When should I use the multi-turn version instead of single-turn?

Use ConversationalGEval when your criteria spans a whole ConversationalTestCase — tone consistency, memory of earlier context, recovery after a bad turn. Plain GEval sees only one input/actual_output pair and can't reason across turns. Neither class accepts the other's test case type.

Does the multi-turn version score each turn separately or the whole conversation?

The whole conversation. ConversationalGEval considers the entire turns history and returns a single score and reason, not one per turn.

When should I reach for G-Eval instead of DAG or a custom metric?

G-Eval for subjective, use-case-specific criteria (correctness, tone, safety) where some non-determinism is fine. Use DAGMetric for deterministic, rule-based scoring, and a custom metric to skip the LLM or combine several metrics.

Should I add every test case parameter to evaluation params?

No. Only include parameters referenced in your criteria/evaluation_steps. Unused params (e.g. expected_output your steps never mention, or MultiTurnParams.TOOLS_CALLED when tool usage isn't part of the criteria) feed the judge irrelevant context and degrade the score.

How do I stop G-Eval from clustering scores in the middle?

Pass a list of Rubrics via the rubric argument to confine the judge to specific score bands (non-overlapping score_ranges within 0-10). For pass/fail, set strict_mode=True to force a binary 1 or 0.

On this page