Eval Modes

In deepeval, the eval mode decides who judges your LLM-as-a-judge metrics and classifiers: an LLM alone, an LLM and Jev together, or Jev alone. Jev is TypeSafe AI's System One model. Instead of generating text, it answers bounded questions (yes/no, a rating on a scale, one option from a set) with calibrated probabilities.

Quick Summary

Every LLM-as-a-judge metric works in steps. It extracts the pieces of your test case it needs to judge (the statements in an answer, the claims it makes, the task an agent was given), decides about each piece, turns those decisions into a score, and writes a reason. The eval mode sets who does each step:

Under llm, each box is an LLM call (except the score, which is an equation). Under hybrid, the decide box goes to Jev. Under system_one, the whole chain collapses into one Jev request over your raw test case.

Setting The Eval Mode

The eval mode is resolved per metric, in this order:

1. The eval_mode argument on the metric.
2. The DEEPEVAL_EVAL_MODE setting, set with the CLI or your environment.
3. The default fallback (llm).

Set it for every metric with the CLI (add --save=dotenv to persist it in .env.local):

deepeval set-eval-mode hybrid
deepeval set-eval-mode system_one
deepeval set-eval-mode llm  # back to the default

Or override it for a single metric, which lets you mix modes in one run:

from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

metrics = [
    FaithfulnessMetric(eval_mode="system_one"),
    AnswerRelevancyMetric(eval_mode="llm"),
]

hybrid and system_one need a TypeSafe AI key: see setting up Jev. Every metric that supports Jev also takes a system_one_model argument to choose the Jev model; it defaults to jev-latest.

LLM-as-a-judge

Under llm, the default, your LLM judge runs every step: it extracts what needs judging from the test case, decides a verdict for each item, and explains the resulting score.

The diagram below shows a typical extract-then-decide metric. Not every metric follows this exact algorithm (some decide in a single step, others score on a rubric), so treat it as an illustration of where the LLM is involved rather than a spec for every metric.

Hybrid

Under hybrid, the LLM extracts and writes the reason, and Jev makes every decision in between. The metric's equation then turns Jev's decisions into a score, the same way it does under llm, so both modes score on the same scale.

A decision takes one of three shapes:

• A verdict per item (claim, statement, opinion, ...): one yes/no question per item.
• A single verdict (per turn, per conversation window, ...): one yes/no question.
• A graded score (Task Completion, Step Efficiency, ...): one rating question on the LLM rubric's levels, mapped onto 0 to 1. When the metric writes its score and reason in the same step, the reason reports Jev's score and confidence.

Yes/no answers become verdicts by thresholding $P(\text{yes})$:

Jev-as-a-judge

Under system_one, Jev runs the whole metric in one request. No LLM is built or called, so no LLM API key is needed.

There is no extract step. Each metric is instead defined as a handful of bounded questions (yes/no, a rating, or a choice) over the raw test case, plus the trace for agentic metrics. Every metric's page lists its questions under "Jev-as-a-judge" in "How Is It Calculated?".

Each answer is mapped onto $[0, 1]$, and the score is their weighted mean:

The reason is built from the same answers rather than generated, so the same answers always produce the same reason.

Confidence Scores

Every metric that Jev judged, in hybrid or system_one, reports metric.confidence: the least decisive of Jev's answers for that measure, from 0 (a coin flip) to 1 (certain). One question on the fence is enough to make the score worth a second look, so the minimum is reported rather than an average.

metric.measure(test_case)
if metric.confidence is not None and metric.confidence < 0.5:
    print("Jev was unsure:", metric.reason)

Ratings and choices carry the confidence Jev reports. A yes/no answer's confidence is $|2p - 1|$, which is what Jev's formula reduces to for two outcomes. Confidence measures how decisive Jev was, not whether it was right, and it never changes the score. It is None under llm.

Which Metrics Support Which Mode

Each metric's page states which eval modes it supports and how it behaves under each. For example:

• GEval: always runs as llm.
• JevEval: always judged by Jev, whatever the eval mode.
• DAGMetric: Jev decides judgement nodes, and system_one runs as hybrid since task nodes need the LLM.
• FaithfulnessMetric: supports all three modes.
• TaskCompletionMetric: supports all three modes, with Jev giving a graded score under hybrid.

Cost

A metric's evaluation_cost is the sum of what its judges charge, so it depends on the eval mode:

Jev charges for input tokens only, not output tokens. If your LLM's pricing is unknown, as with a custom model, evaluation_cost is None under llm and hybrid.