JevEval

Q: How is JevEval different from G-Eval?

GEval asks an LLM to draft evaluation steps from a criteria and then generate a score, and uses token probabilities to smooth that number. JevEval has no generative judge: you write the questions, Jev returns calibrated probabilities, and the score is a fixed weighted mean. Use G-Eval when you want the LLM to figure out the logic; use JevEval when you know what you want decided and want it decided the same way every run.

Q: When should I use Choice instead of Score?

Use a Score when the outcomes sit on a line from worst to best. Use a Choice when they do not: two options can share a credit, and an option can mean "this question does not apply" by setting its credit to .

Q: How is a Choice question turned into a score?

Jev returns a probability per option. The options are summed first: if they hold half or more of the mass the question is dropped. Otherwise the remaining options are renormalised and multiplied by their credits. In the worked example that is (0.05·1.0 + 0.05·1.0 + 0.25·0.5 + 0.60·0.0) / 0.95 = 0.237.

Q: Can I run JevEval without any LLM?

JevEval never uses one. The score is one Jev request plus arithmetic, and the reason is built in code from Jev's answers and their confidence, so no LLM provider has to be configured. Set if you do not want the reason text at all.

Q: Can I use JevEval on a conversation?

Yes, with ConversationalJevEval. It takes a ConversationalTestCase, puts the turns into the state as a turns list, and scores with the same three question types and the same equation. Switch to the Multi-Turn tab in any code block above.

Q: Do I have to list CONTENT and ROLE in evaluation params for a conversation?

No. Every question is about the conversation, so MultiTurnParams.CONTENT and MultiTurnParams.ROLE are added automatically. List only the extra fields your questions need, such as SCENARIO, EXPECTED_OUTCOME, CHATBOT_ROLE, RETRIEVAL_CONTEXT or TOOLS_CALLED.

Q: How do I handle behaviour the user never triggered in a conversation?

Give the Choice an option with credit , such as never_asked. When Jev puts half or more of the probability on it, the question drops out of the score for that conversation instead of penalising the chatbot for something that never came up.

Jev-as-a-judge

Custom

Single-turn

Multi-turn

JevEval is a custom metric that scores an LLM interaction as the weighted mean of the answers to bounded questions you write about it. Each question is answered by Jev, a System One model, as a calibrated probability rather than generated text; the probability is the question's value, and the metric score is the weighted mean of those values.

There is no chain-of-thought, no "give this a score from 1 to 10", and no JSON to recover. No LLM is involved at any point: even the reason is deterministic text built from Jev's answers and their confidence, so the same test case always produces the same reason.

Usage

First, if you haven't already, install TypeSafe AI's SDK and save your Jev API key:

pip install typesafe-sdk
export TYPESAFE_API_KEY=<your-typesafe-api-key>

Instantiate the metric with the test case fields your questions talk about, and the questions themselves:

Here we build a custom Tool Faithfulness metric that checks whether the LLM hallucinated on top of what its tools actually returned:

from deepeval.metrics.jev_eval import Noul, Score, Choice
from deepeval.test_case import SingleTurnParams
from deepeval.metrics import JevEval

tool_faithfulness = JevEval(
    name="Tool Faithfulness",
    evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.TOOLS_CALLED],
    questions=[
        Noul("Every fact and figure in actual_output appears in the output of a tool in tools_called.", weight=2),
        Noul("actual_output reports every value returned in tools_called accurately."),
        Score(
            "How much of actual_output is grounded in the outputs in tools_called?",
            levels=["Fabricated", "Mostly fabricated", "Mostly grounded", "Fully grounded"],
        ),
        Choice(
            "What did actual_output do with information the tools did not return?",
            options={"left_it_out": 1.0, "flagged_it_as_unknown": 1.0, "hedged_it": 0.5, "stated_it_as_fact": 0.0, "nothing_missing": None},
        ),
    ],
)

Here we build a custom Refund Handling metric that checks how a support chatbot handled a refund request across a conversation:

from deepeval.metrics.jev_eval import Noul, Score, Choice
from deepeval.test_case import MultiTurnParams
from deepeval.metrics import ConversationalJevEval

refund_handling = ConversationalJevEval(
    name="Refund Handling",
    evaluation_params=[MultiTurnParams.SCENARIO, MultiTurnParams.EXPECTED_OUTCOME],
    questions=[
        Noul("The assistant verifies the order before acting on the refund."),
        Score(
            "How fully is the user's request settled by the end of turns?",
            levels=["Not addressed", "Partly settled", "Fully resolved"],
        ),
        Choice(
            "What did the assistant do about the amount the user asked for?",
            options={"stated_it": 1.0, "deferred_it": 0.5, "ignored_it": 0.0, "never_asked": None},
        ),
    ],
)

There are THREE mandatory and SEVEN optional parameters when creating a JevEval metric:

name: name of custom metric.
evaluation_params: a list of type SingleTurnParams (or MultiTurnParams for multi-turn). These fields are the state Jev sees; a question can only refer to fields listed here. For single-turn JevEval, omit it to evaluate the trajectory instead.
questions: a list of Noul, Score and Choice questions. Each has a weight (default 1.0).
[Optional] system_one_model: the Jev model to use, as a string or a DeepEvalBaseSystemOneModel. Defaulted to jev-latest.
[Optional] threshold: the passing threshold. Can also be set to None to run the metric in score-only mode. Defaulted to 0.5.
[Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 if every applicable question is answered in its best possible way, 0 otherwise. It also overrides the current threshold and sets it to 1. See strict mode. Defaulted to False.
[Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
[Optional] verbose_mode: a boolean which when set to True, prints every question's probabilities and value to the console. Defaulted to False.
[Optional] flaky: a boolean which when set to True, marks the metric as flaky. Defaulted to False.
[Optional] include_reason: a boolean which when set to True, attaches a reason listing each question's outcome in words, the probability and confidence behind it, and how the score follows. The reason is built in code from Jev's answers, so it costs nothing and never varies between runs. Defaulted to True.

As with every deepeval metric, JevEval returns a score from 0 - 1 and is successful when that score is equal to or greater than threshold. You can access score, reason and the per-question score_breakdown:

from deepeval.test_case import LLMTestCase, ToolCall
from deepeval import evaluate
...

test_case = LLMTestCase(
    input="What's the weather in Paris right now?",
    actual_output="It's 18°C and sunny in Paris, with a light breeze and around 40% humidity.",
    tools_called=[
        ToolCall(
            name="get_weather",
            input_parameters={"city": "Paris"},
            output={"temp_c": 18, "condition": "sunny"},
        )
    ],
)

# To run metric as a standalone
# tool_faithfulness.measure(test_case)
# print(tool_faithfulness.score, tool_faithfulness.reason)
# print(tool_faithfulness.score_breakdown)

evaluate(test_cases=[test_case], metrics=[tool_faithfulness])

from deepeval.test_case import ConversationalTestCase, Turn
from deepeval import evaluate
...

convo_test_case = ConversationalTestCase(
    scenario="A customer asks for a refund on a late order.",
    expected_outcome="The refund amount is confirmed.",
    turns=[
        Turn(role="user", content="My order arrived a week late. I want a refund."),
        Turn(role="assistant", content="Sorry about that. What is your order number?"),
        Turn(role="user", content="It's 4471. How much do I get back?"),
        Turn(role="assistant", content="You'll receive a refund within 5-7 days."),
        Turn(role="user", content="ok thanks"),
    ],
)

# To run metric as a standalone
# refund_handling.measure(convo_test_case)
# print(refund_handling.score, refund_handling.reason)
# print(refund_handling.score_breakdown)

evaluate(test_cases=[convo_test_case], metrics=[refund_handling])

Evaluating the trajectory

When you leave out evaluation_params, single-turn JevEval sends your AI agent's trace to Jev instead of test case fields. The state is {"trace": ...}: the span tree, where each span keeps its name, type, inputs, outputs and tool calls, nested under children. Token counts, costs and prompts are dropped to save Jev's context, so questions can't ask about them.

Write the questions about the agent's run, and refer to spans by name where you can:

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics.jev_eval import Noul, Choice
from deepeval.tracing import observe
from deepeval.metrics import JevEval

tool_use = JevEval(
    name="Tool Use",
    questions=[
        Noul("The agent called the order_lookup tool before answering.", weight=2),
        Noul("The order_id passed to order_lookup matches the order the user asked about."),
        Choice(
            "How does the agent's final output relate to what order_lookup returned?",
            options={"consistent": 1.0, "partly_consistent": 0.5, "contradicts_it": 0.0, "no_tool_called": None},
        ),
    ],
)

@observe(type="tool")
def order_lookup(order_id: str):
    return {"status": "in_transit"}

@observe(type="agent")
def support_agent(input: str):
    return f"Your order is {order_lookup('1234')['status']}."

dataset = EvaluationDataset(goldens=[Golden(input="Where is my order #1234?")])
for golden in dataset.evals_iterator(metrics=[tool_use]):
    support_agent(golden.input)

Passed to evals_iterator, the metric sees the full trace. Attached to a single component through that component's metrics, it sees only that span and its children.

Question Types

JevEval has three question types, drawn directly from Jev's decision primitives. Each maps its answer onto a value $v \in [0, 1]$ with a fixed rule.

Question type	Best for	Example
`Noul`	A yes/no check: one proposition that is either true or false	"Every figure in `actual_output` appears in a tool output."
`Score`	A judgement of degree along an ordered scale, worst to best	"How much of `actual_output` is grounded?" Fabricated → Fully
`Choice`	Picking one of several distinct behaviours, some of which may not apply	"What did it do with data the tools never returned?"

Noul

A Noul is one proposition about the state. Jev returns $P(\text{true})$ , and that probability is the value:

v_{\text{Noul}} = P(\text{true})

from deepeval.metrics.jev_eval import Noul

Noul("Every fact and figure in actual_output appears in the output of a tool in tools_called.", weight=2)

Use Nouls for a checklist: several independent checks, each calibrated, each weighted by how much you care. Noul also accepts optional true / false descriptions that sharpen what counts as either side.

Score

A Score is one holistic judgement over ordered descriptive levels, worst first. Jev returns a probability for every level; the value is the expected level position, normalised so the top level is 1:

v_{\text{Score}} = \frac{1}{n-1}\sum_{k=0}^{n-1} k \, P(k) \qquad \text{for } n \text{ ordered levels, worst first}

from deepeval.metrics.jev_eval import Score

Score(
    "How much of actual_output is grounded in the outputs in tools_called?",
    levels=["Fabricated", "Mostly fabricated", "Mostly grounded", "Fully grounded"],
)

Use a Score when "between two levels" is meaningful. Jev accepts between 2 and 10 levels.

Choice

A Choice is one selection from an unordered closed set. You attach a credit in [0, 1] to every option, or None to mark an option that means "this question does not apply here". The value is the credit-weighted probability over the applicable options:

v_{\text{Choice}} = \frac{\displaystyle\sum_{o \,:\, c_o \neq \varnothing} P(o)\, c_o}{\displaystyle\sum_{o \,:\, c_o \neq \varnothing} P(o)} \qquad \text{for options } o \text{ with credits } c_o \in [0, 1] \cup \{\varnothing\}

from deepeval.metrics.jev_eval import Choice

Choice(
    "What did actual_output do with information the tools did not return?",
    options={"left_it_out": 1.0, "flagged_it_as_unknown": 1.0, "hedged_it": 0.5, "stated_it_as_fact": 0.0, "nothing_missing": None},
)

If the None options collect half or more of the probability, the question is dropped from the metric for that test case: it did not apply, so it neither helps nor hurts.

Credits do not have to be monotone, which is what a Score cannot express. Above, left_it_out and flagged_it_as_unknown are two very different behaviours that deserve the same full credit, and nothing_missing means the question should not count at all.

Writing questions about a conversation

Multi-turn JevEval uses the same three question types. What changes is what the questions are about:

Write questions about turns as a whole ("by the end of turns", "at any point in turns") or about a role ("the assistant", "the user").
Reference conversation-level fields by name: "the scenario", "matches expected_outcome", "stays in chatbot_role".
Use a Choice with a None option for behaviours that may never have come up in this particular conversation, so the question drops out instead of penalising the chatbot for something the user never asked.

How Is It Calculated?

The JevEval score is calculated according to the following equation:

\text{JevEval} = \frac{\displaystyle\sum_{i \in \mathcal{A}} w_i \, v_i}{\displaystyle\sum_{i \in \mathcal{A}} w_i}

Where $w_i$ is the weight of question $i$ , $v_i \in [0, 1]$ is the value Jev's answer maps onto, and $\mathcal{A}$ is the set of questions that applied to this test case.

The metric first builds a JSON state from the fields listed in evaluation_params, then sends the state and every question to Jev in a single request. For single-turn the state is the listed fields of the test case; for multi-turn it is the entire conversation plus the conversation-level fields you listed. Jev is a System One model: it answers bounded questions with calibrated probabilities rather than generating language. Jev returns a probability distribution per question; no text is generated. Each distribution becomes a value $v_i$ using the rule for its question type: $v_{\text{Noul}}$ , $v_{\text{Score}}$ , or $v_{\text{Choice}}$ .

A Choice question applies only if its None-credited options hold less than half the probability:

i \in \mathcal{A} \iff \sum_{o \,:\, c_o = \varnothing} P(o) < 0.5

Noul and Score questions always apply. If $\mathcal{A}$ is empty, the score is 1: nothing applicable was left to fail. For conversations this is how you keep a chatbot from being penalised for behaviour the user never triggered.

Strict mode

With strict_mode=True, the weighted mean is replaced by an all-or-nothing check. Every applicable question must be answered in its best possible way, decided from Jev's probabilities alone:

\text{JevEval}_{\text{strict}} = \begin{cases} 1 & \text{if every } i \in \mathcal{A} \text{ passes} \\ 0 & \text{otherwise} \end{cases}

Where a question passes when:

Noul: $P(\text{true}) \ge 0.5$
Score: Jev's most probable level is the top level
Choice: Jev's most probable applicable option carries a credit of 1.0

threshold is set to 1, and each entry of score_breakdown gains a passed flag so you can see which question fell short. Questions that did not apply to this test case are skipped, as usual.

With include_reason=True, a reason is then built from the same answers: one line per question with its outcome in words, the probability and confidence behind it, and a closing line showing how the score follows. Nothing is generated, so there is no LLM call at any point.

Example

Let's trace the Tool Faithfulness metric from Usage on the Paris weather test case end to end. Every number below is one you can read back from score_breakdown. The same trace for the multi-turn Refund Handling metric is in the JevEval examples guide.

Build the state

evaluation_params decides what Jev gets to see. The three fields are pulled off the LLMTestCase into one JSON object, with each ToolCall serialised as structured data so Jev can read its output directly:

{
  "test_case": {
    "input": "What's the weather in Paris right now?",
    "actual_output": "It's 18°C and sunny in Paris, with a light breeze and around 40% humidity.",
    "tools_called": [
      {
        "name": "get_weather",
        "type": "FUNCTION",
        "input_parameters": {"city": "Paris"},
        "output": {"temp_c": 18, "condition": "sunny"}
      }
    ]
  }
}

This is why the questions say actual_output and tools_called: they name keys in this object. Had a question mentioned retrieval_context, Jev would simply not have it.

Translate the questions

Each question becomes one entry in a single Jev request, keyed q_0 to q_3:

q_0, q_1: yes/no questions on the two Noul statements.
q_2: a score question over the four Score levels.
q_3: a choice question over the five option names only. The credits (1.0, 1.0, 0.5, 0.0, None) are never sent; they are deepeval-side bookkeeping for the next step.

One decide() call sends the state and all four questions. Jev evaluates them in parallel and returns probabilities, no text.

What Jev returns

An illustrative answer for this test case:

q_0  P(true) = 0.30      # "light breeze" and "40% humidity" are in no tool output
q_1  P(true) = 0.90      # 18°C and sunny match temp_c and condition exactly
q_2  score = 1.7, probabilities {0: 0.05, 1: 0.30, 2: 0.55, 3: 0.10}, confidence 0.55
q_3  choice = stated_it_as_fact,
     probabilities {left_it_out: 0.05, flagged_it_as_unknown: 0.05, hedged_it: 0.25, stated_it_as_fact: 0.60, nothing_missing: 0.05},
     confidence 0.6

Map each answer to a value

This is where the three primitives become comparable.

q_0 (Noul, weight 2) and q_1 (Noul, weight 1) are already values:

v_0 = 0.30 \qquad v_1 = 0.90

q_2 (Score) has four levels, so $n - 1 = 3$ :

v_2 = \frac{0(0.05) + 1(0.30) + 2(0.55) + 3(0.10)}{3} = \frac{1.7}{3} = 0.567

Landing between "Mostly fabricated" and "Mostly grounded" is the point: the expected position is already a soft score.

q_3 (Choice) has nothing_missing marked None. Its mass is $0.05 < 0.5$ , so the question applies; the tool never returned wind or humidity. Renormalise over the other four and weight by credit:

v_3 = \frac{0.05 \cdot 1.0 + 0.05 \cdot 1.0 + 0.25 \cdot 0.5 + 0.60 \cdot 0.0}{0.05 + 0.05 + 0.25 + 0.60} = \frac{0.225}{0.95} = 0.237

Had the response stuck to "18°C and sunny" and Jev put 0.9 on nothing_missing, q_3 would drop out entirely and the other three questions would decide the score alone.

Weighted mean

\text{JevEval} = \frac{2 \cdot 0.30 + 1 \cdot 0.90 + 1 \cdot 0.567 + 1 \cdot 0.237}{2 + 1 + 1 + 1} = \frac{2.304}{5} = 0.461

tool_faithfulness.score is 0.461 and success is False against the default threshold of 0.5. The response got the tool's own values right, but the "every fact appears in a tool output" Noul carried weight 2, so the invented breeze and humidity pulled the score down more than any other single question could. Weight is the only place your "how much do I care" enters the math.

Read the breakdown

score_breakdown keeps every intermediate so you can see why:

[
  {"question": "Every fact and figure in actual_output appears in the output of a tool in tools_called.", "type": "noul", "weight": 2.0, "value": 0.30, "applicable": True, "probabilities": {"true": 0.30, "false": 0.70}, "confidence": 0.40},
  {"question": "actual_output reports every value returned in tools_called accurately.", "type": "noul", "weight": 1.0, "value": 0.90, "applicable": True, "probabilities": {"true": 0.90, "false": 0.10}, "confidence": 0.80},
  {"question": "How much of actual_output is grounded in the outputs in tools_called?", "type": "score", "weight": 1.0, "value": 0.567, "applicable": True, "probabilities": {"Fabricated": 0.05, "Mostly fabricated": 0.30, "Mostly grounded": 0.55, "Fully grounded": 0.10}, "confidence": 0.55},
  {"question": "What did actual_output do with information the tools did not return?", "type": "choice", "weight": 1.0, "value": 0.237, "applicable": True, "probabilities": {"left_it_out": 0.05, "flagged_it_as_unknown": 0.05, "hedged_it": 0.25, "stated_it_as_fact": 0.60, "nothing_missing": 0.05}, "confidence": 0.6},
]

Reason

With include_reason=True, a reason is built from the answers above. It names the judge and the least decisive answer, then walks through each question with its outcome in words and the numbers behind it (a Noul's confidence is |2p - 1|, the same statistic Jev reports for Choice and Score):

Decided by jev-latest, minimum confidence 0.40.
1. Every fact and figure in actual_output appears in the output of a tool in tools_called. -> likely fails (P(yes)=0.30, confidence=0.40, weight=2)
2. actual_output reports every value returned in tools_called accurately. -> clearly holds (P(yes)=0.90, confidence=0.80)
3. How much of actual_output is grounded in the outputs in tools_called? -> "Mostly grounded" (expected level=0.57 of 1.00, confidence=0.55)
4. What did actual_output do with information the tools did not return? -> "stated_it_as_fact" (P=0.60, confidence=0.60)
Score: 0.46 (weighted mean of 4 applicable questions).

Nothing here is generated: the same answers always produce the same reason, and the whole metric ran with zero LLM tokens whether or not include_reason is set. The confidence on the metric (0.40 here) is the minimum across every question, so you can filter or flag results where Jev was not decisive.

For more ready-made question sets, single-turn and multi-turn, see the JevEval examples guide.

FAQs

How is JevEval different from G-Eval?

GEval asks an LLM to draft evaluation steps from a criteria and then generate a score, and uses token probabilities to smooth that number. JevEval has no generative judge: you write the questions, Jev returns calibrated probabilities, and the score is a fixed weighted mean. Use G-Eval when you want the LLM to figure out the logic; use JevEval when you know what you want decided and want it decided the same way every run.

When should I use Choice instead of Score?

Use a Score when the outcomes sit on a line from worst to best. Use a Choice when they do not: two options can share a credit, and an option can mean "this question does not apply" by setting its credit to None.

How is a Choice question turned into a score?

Jev returns a probability per option. The None options are summed first: if they hold half or more of the mass the question is dropped. Otherwise the remaining options are renormalised and multiplied by their credits. In the worked example that is (0.05·1.0 + 0.05·1.0 + 0.25·0.5 + 0.60·0.0) / 0.95 = 0.237.

Can I run JevEval without any LLM?

JevEval never uses one. The score is one Jev request plus arithmetic, and the reason is built in code from Jev's answers and their confidence, so no LLM provider has to be configured. Set include_reason=False if you do not want the reason text at all.

Can I use JevEval on a conversation?

Yes, with ConversationalJevEval. It takes a ConversationalTestCase, puts the turns into the state as a turns list, and scores with the same three question types and the same equation. Switch to the Multi-Turn tab in any code block above.

Do I have to list CONTENT and ROLE in evaluation params for a conversation?

No. Every question is about the conversation, so MultiTurnParams.CONTENT and MultiTurnParams.ROLE are added automatically. List only the extra fields your questions need, such as SCENARIO, EXPECTED_OUTCOME, CHATBOT_ROLE, RETRIEVAL_CONTEXT or TOOLS_CALLED.

How do I handle behaviour the user never triggered in a conversation?

Give the Choice an option with credit None, such as never_asked. When Jev puts half or more of the probability on it, the question drops out of the score for that conversation instead of penalising the chatbot for something that never came up.

On this page