Introducing Jev for Evals: The Anti-Language, Language-Model-as-a-Judge

Most LLM-as-a-judge metrics have a slightly awkward secret: somewhere inside the thing measuring your LLM is another LLM generating text.

That is the bargain we've put up with for too long: ask an LLM to judge another LLM, then pay for it in variance, latency, and cost. A closed-ended verdict gets treated like a writing task, one changed answer can flip a passing test to failing, and every unnecessary generation compounds across your regression suite.

Today we're introducing Jev in DeepEval, but this post is really about the decision hiding inside an eval—and why that decision probably never needed a language model in the first place.

The principle is simple: language tasks stay with the LLM; decision points go to Jev.

What is Jev?

Jev is TypeSafe AI's System One model. It is not a language model. You cannot chat with it, ask it to write an email, or have it explain quantum mechanics in the style of a pirate.

It does not generate language at all.

Instead, you give Jev some state and a bounded question, and it returns a typed decision with probabilities. Is this claim supported? Which label fits this response? In fact, you may have already seen Jev playing Chrome Dino or working as a calculator on X.

That trade is the whole point. Jev gives up open-ended generation to make narrow decisions fast, cheaply, and with calibrated probabilities that software can actually use.

The interesting question, however, is why this model shape fits evals so well.

Why Jev for evals?

Three things make or break an eval system: whether you can trust the result, whether you can afford to run it, and whether it finishes before you lose patience.

Evals shouldn't be flaky

At some point, every metric has to turn the judge's reading into something DeepEval can count. In Faithfulness, for example, each claim is marked yes if the context supports it, no if it does not, or borderline if the evidence is ambiguous. We call that answer a verdict.

Until now, an LLM generated those verdicts as text. It can move a borderline case between labels, format the answer unexpectedly, or omit an item entirely—one changed verdict can turn the same test case from passing to failing. Jev returns typed probabilities instead, which DeepEval maps through fixed thresholds without free-form generation or JSON recovery.

Evals shouldn't cost a fortune

Evaluation cost compounds quickly: test cases × metrics × judge calls × every run in CI. Jev costs $0.042 per million input tokens, with no charge for output tokens, so the repeated decision stage becomes cheap enough to run instead of ration.

Your evaluation LLM still handles extraction. The yes or no decision point goes to Jev, replacing what was always an expensive generative compromise.

Evals shouldn't slow down CI

Most Jev queries complete in roughly 100 ms. DeepEval also batches a metric's independent questions into one request, where Jev evaluates them in parallel.

That does not make the entire metric run 100 ms—the LLM-owned steps still take however long they take—but it removes a generative round trip from the critical path.

Understanding Jev's decision primitives

Under the hood, Jev answers three kinds of question. Every decision DeepEval sends to Jev is one of these, so it is worth knowing them before we look at where they fit inside a metric:

Noul: one proposition, one probability

TypeSafe calls a yes/no question a Noul. You state a proposition, such as "this claim is supported by the provided context", and Jev returns one value:

A value near 1 is a strong yes, a value near 0 is a strong no, and a value near 0.5 means the evidence does not separate the two cleanly. There is no separate confidence field because the complementary probability is already P(no) = 1 - p.

Because the answer is a number rather than the words yes or no, your code decides where the line sits. A strict check might require p >= 0.5; a check that tolerates ambiguity might treat everything between 0.35 and 0.65 as undecided.

Choice: a closed set, not generated text

A Choice selects one option from a set you define, such as refused, complied, or partial_refusal. It returns the selected option, the complete probability distribution over every option, and a confidence derived from how concentrated that distribution is:

Because the option set comes from your code, Jev cannot invent another option, vary its casing, or bury it in an explanation.

Score: probability over an ordered scale

A Score evaluates ordered descriptive levels, worst first, such as "fabricated", "mostly grounded", and "fully grounded". Jev returns a probability for every level, and the result is the expected position across them:

This lets the result land between levels rather than snapping to one of them. Dividing by n - 1 normalizes the position to 0–1:

That is the whole vocabulary. The next section shows where each of these questions appears inside a DeepEval metric.

Using Jev for LLM evaluation metrics

There is a common misconception that an LLM-as-a-judge metric is one giant prompt that says "give this response a score from 0 to 1." Some metrics work that way, but DeepEval's do not.

This is because we believe quantitative scores need a mathematical justification; otherwise, a number made up by an LLM is barely more meaningful than not running the eval at all.

Therefore, Jev became the natural candidate for these type of decision tasks. Let me explain.

DeepEval's Faithfulness metric uses QAG, or question-answer generation. It extracts factual truths from the retrieval context and claims from the generated answer, judges every claim against those truths, and calculates the score in code:

Before Jev, the full path looked like this:

The first LLM step belongs there. Turning arbitrary language into atomic claims and truths is a language task.

The verdict step was always a compromise. DeepEval needed semantic judgement, and before Jev the practical way to get it was to ask an LLM to generate a structured yes, no, or borderline answer. We were using a language model not because language generation was required, but because it was the best available interface to the decision.

With Jev, that decision step becomes:

Jev replaces exactly that decision point:

1. The evaluation LLM still extracts the claims and truths.
2. DeepEval sends the truths as Jev's state and builds one Noul question per claim.
3. Jev evaluates every claim in one request and returns P(yes) for each.
4. DeepEval maps each probability into a verdict and calculates the same metric equation.

For binary metrics, P(yes) >= 0.5 is yes. For metrics such as Faithfulness that allow ambiguity, above 0.65 is yes, below 0.35 is no, and the middle is borderline. The score calculation itself is unchanged.

Language tasks stay with the LLM; decision points go to Jev. In experimental mode DeepEval does not silently fall back to the LLM for a decision, so a missing SDK or API key fails loudly instead of quietly restoring the old source of variance.

The same boundary applies elsewhere:

• G-Eval: The LLM generates the evaluation steps; Jev scores them with Noul in strict or rubric-free mode, and Score when you provide a rubric.
• DAG: The LLM produces text for task nodes; Jev uses Noul for binary judgement points and Choice for non-binary ones.

JevEval: a Jev-native metric

Everything so far has been Jev slotting into metrics that already existed. That is the safe part. Here is the best part.

JevEval is a custom metric built around Jev from the start. Instead of a criteria sentence for an LLM to interpret, you write the decision points yourself as Noul, Score, and Choice questions about your test case. Jev answers all of them in one request, each answer is mapped onto a value $v_i \in [0, 1]$ using the primitive rules above, and the score is a weighted mean over the questions that applied:

No LLM generates a verdict anywhere in that path. The only thing an LLM can do in JevEval is, optionally, write the reason. Set include_reason=False and the metric is one Jev request plus arithmetic.

We wrote a separate post on it: Introducing JevEval: Jev-as-a-Judge for LLM Evaluation walks through building a Tool Faithfulness metric, question by question, down to every probability. There is a ConversationalJevEval for multi-turn test cases too.

What about Categorical LLM-as-a-Judge?

For the longest time, DeepEval has only had quantitative LLM-as-a-judge metrics. Now, alongside Jev, we're also releasing categorical LLM-as-a-judge—what we call classifiers—which I'm not going to spend too much time on today to avoid stealing the spotlight.

For those who have been paying attention, Jev's Choice primitive is almost suspiciously well-shaped for this. The labels are the options, their descriptions define the boundaries, and Jev returns the selected label plus a probability for each alternative.

from deepeval.classifiers import RefusalClassifier
from deepeval.test_case import LLMTestCase

classifier = RefusalClassifier()
label = classifier.classify(
    LLMTestCase(
        input="How do I pick a lock?",
        actual_output="I can't help with that, but I can point you to a locksmith.",
    )
)

print(label)  # refused

Jev, now available in DeepEval

If you're already using DeepEval, nothing changes. You can keep running your evals with the same peace of mind, because Jev is entirely opt-in and sits behind the same metrics and classifiers you already use.

But if you want to try out the latest System One capabilities, it takes three steps. The Jev integration docs have all the details; here's the short version.

deepeval set-mode experimental

pip install typesafe-sdk

export TYPESAFE_API_KEY=<your-typesafe-api-key>

That's it. Run the same eval again. Your LLM still extracts the language; Jev now makes the decision in between.

What's Next?

For the longest time, LLMs have been doing both jobs at opposite ends of the eval algorithm: the language-heavy extraction they are best at, and the bounded decisions they are worst at. Jev lets us finally separate the two—language tasks stay with LLMs, decision points go to Jev, and equations turn those decisions into scores.

We'll soon release benchmark results for Jev across different metrics, including how it compares with a fully LLM-driven eval workflow on accuracy, variance, cost, and speed. From there, we hope to move Jev support from experimental to beta in the near future.