TypeSafe AI (Jev)

deepeval supports using TypeSafe AI's Jev, a System One model, as the judge for its metrics and classifiers under the hybrid and system_one eval modes, and for JevEval. To get started, you'll need to install the TypeSafe AI SDK and set up your API key.

Setting Up Your API Key

Install the TypeSafe AI SDK, then provide your TYPESAFE_API_KEY in the CLI:

pip install typesafe-sdk
export TYPESAFE_API_KEY=<your-typesafe-api-key>

Alternatively, if you're working in a notebook environment (e.g., Jupyter or Colab), set your TYPESAFE_API_KEY in a cell:

%env TYPESAFE_API_KEY=<your-typesafe-api-key>

You can also save it with deepeval's CLI, which accepts --model and --cost-per-input-token too (run deepeval unset-typesafe to remove it):

deepeval set-typesafe --prompt-api-key --save=dotenv

In Code

To use Jev for deepeval metrics, define a TypeSafeModel and pass it as the metric's system_one_model. By default, the model is set to jev-latest.

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.models import TypeSafeModel

model = TypeSafeModel(model="jev-latest")
answer_relevancy = AnswerRelevancyMetric(
    system_one_model=model,
    eval_mode="system_one",
)

There are ZERO mandatory and THREE optional parameters when creating a TypeSafeModel:

[Optional] model: A string specifying which Jev model to use. Defaults to TYPESAFE_MODEL_NAME if not passed; falls back to jev-latest if unset.
[Optional] api_key: A string specifying your TypeSafe AI API key. Defaults to TYPESAFE_API_KEY if not passed; raises an error at runtime if unset.
[Optional] cost_per_input_token: A float specifying the cost for each input token for the provided model. Defaults to TYPESAFE_COST_PER_INPUT_TOKEN, then to deepeval's pricing for known Jev models. Jev does not charge for output tokens.

Extra **kwargs passed to TypeSafeModel(...) are forwarded to the underlying TypeSafeClient.

With TYPESAFE_API_KEY set, pass the name of your desired Jev model as the metric's system_one_model, or leave it out to use TYPESAFE_MODEL_NAME (falling back to jev-latest):

from deepeval.metrics import AnswerRelevancyMetric

answer_relevancy = AnswerRelevancyMetric(
    system_one_model="jev-latest",
    eval_mode="system_one",
)

To use Jev in every metric without passing eval_mode, set the eval mode once with deepeval set-eval-mode hybrid or deepeval set-eval-mode system_one.

Available Jev Models

Below is a list of commonly used Jev models:

jev-latest (alias of the current stable release)
jev-preview
jev-1.13.0

Setting Up Your API Key

In Code

Available Jev Models

On this page