DeepEval for TypeScript, now fully open-source

Two months ago I wrote about why we put TypeScript inside DeepEval's Python monorepo instead of giving it its own repo. At the time it was still a client wrapper around Confident AI — it couldn't run a single metric.

Today it can run all of them. DeepEval for TypeScript is out in beta:

npm install -D deepeval

The headline isn't the metrics though. It's where they run: in CI/CD, as a gate on your pull requests.

Evals belong in your Typescript CI pipeline

The reason I keep pushing this is that an eval you run by hand is a nice number, and an eval that runs on every PR is a decision. That's the whole point of an eval harness — a regression in your agent should block a merge exactly like a regression in your business logic does, without anyone remembering to check.

For that to happen, evals have to live where your pipeline already looks: your test suite. In Python that's Pytest. In TypeScript it's Vitest, and the surface is a single matcher — toPass() — on top of a test file you'd recognize without ever having used DeepEval:

import { EvaluationDataset, Golden } from "deepeval/dataset";
import { AnswerRelevancyMetric } from "deepeval/metrics";
import { LLMTestCase } from "deepeval/test-case";
import { it, expect } from "vitest";
import "deepeval/vitest";

const dataset = new EvaluationDataset({
  goldens: [new Golden({ input: "What is pi rounded to 2 decimal places?" })],
});

it.each(dataset.goldens as Golden[])(
  "answers correctly #%$",
  async (golden) => {
    const testCase = new LLMTestCase({
      input: golden.input,
      actualOutput: await yourLlmApp(golden.input),
    });
    await expect(testCase).toPass([new AnswerRelevancyMetric()]);
  }
);

Then run it:

npx deepeval test run llm_app.test.ts

Plain vitest works too, but you'd be leaving most of the value on the table. npx deepeval test run captures a trace per test, caches metric results so a re-run doesn't re-bill you, and gives you the same flags Python users have — --official, -i/--identifier, --max-concurrent, -c/--use-cache, --ignore-errors.

That one command is the whole integration story. It exits non-zero when a metric falls below its threshold, so any CI provider that runs a shell step already knows what to do with it:

name: LLM App `deepeval` Tests

on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - run: npm ci

      - name: Run evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: npx deepeval test run llm_app.test.ts

Note what isn't in there: no CONFIDENT_API_KEY, no hosted service, no vendor in the critical path of your merge queue. Give it a judge key and the whole thing runs on the runner. Add the key later if you want shared reports and regression tracking across commits, and add --official on main to mark the baseline that future runs get compared against. The full walkthrough is in unit testing in CI/CD.

Every metric, open-source

This is the part I care most about, because a second language that only ships the easy half of the metric library is worse than no second language at all.

47 of DeepEval's 49 metrics are ported and open-source in TypeScript. Not a curated subset — G-Eval with log-prob weighted scoring, DAG decision graphs, the multi-turn suite, the MCP metrics, the multimodal ones, arena comparisons. The only two that haven't landed are AgentLoopDetectionMetric and ToolPermissionMetric, and they'll be in shortly.

They agree with Python because they aren't rewritten from memory. Both SDKs compile the same language-neutral prompt templates — one JSON bundle of judge prompts, shared by the two implementations. Only the orchestration and the output schema (Pydantic in Python, Zod in TypeScript) are written per-language. So when we change a judge prompt, both languages change together or neither does.

import { GEval } from "deepeval/metrics";

const correctness = new GEval({
  name: "Correctness",
  criteria: "Does the output match the expected answer?",
  evaluationParams: ["input", "actualOutput", "expectedOutput"],
  threshold: 0.5,
});

The API is idiomatic TypeScript rather than transliterated Python — an options object instead of keyword arguments, camelCase, and measure() is always async (there's no sync/a_measure split, everything underneath is already async). Full list on the metrics page.

The models that judge them

An LLM-as-a-judge metric is only as portable as the providers it can run on, so the model layer came over too. TypeScript ships 13 model integrations.

Two notes. LiteLLM is Python-only and always will be — AISDKModel is the TypeScript answer to "route my judge through anything." And the default judge model is generated from Python's DEFAULT_MODELS, so an unconfigured metric lands on the same model in both languages.

Set one once from the CLI and every metric picks it up:

npx deepeval set-anthropic --model claude-sonnet-4-5

Tracing, and evals on the trace

Scoring a final string is fine for a RAG pipeline. It isn't enough for an agent, where the interesting failures are in the trajectory — the tool it shouldn't have called, the plan it abandoned, the four steps it took to do one thing.

So TypeScript gets tracing, with 6 framework integrations that turn your agent's execution into a span tree with no rewriting:

Mastra and the Vercel AI SDK are the fun ones — they have no Python counterpart at all, so for once TypeScript is ahead.

The reason this matters for CI/CD is that instrumenting your agent and gating on it are the same piece of work, not two. Once the app is traced, you hand toPass() a golden and a task, and it runs your agent, captures the trace, and scores the whole trajectory:

import { TaskCompletionMetric, StepEfficiencyMetric } from "deepeval/metrics";
import { DeepEvalCallbackHandler } from "deepeval/integrations/langchain";
import { it, expect } from "vitest";
import "deepeval/vitest";

it.each(dataset.goldens as Golden[])(
  "completes the task #%$",
  async (golden) => {
    await expect(golden).toPass(
      [new TaskCompletionMetric(), new StepEfficiencyMetric()],
      {
        task: (g) =>
          agent.invoke(
            { messages: [{ role: "user", content: g.input }] },
            { callbacks: [new DeepEvalCallbackHandler({})] }
          ),
      }
    );
  }
);

You can also attach metrics to individual spans instead of the whole trace — component-level evals, where a retriever gets contextual precision and a tool call gets argument correctness, in the same test run.

The CLI came over too

Same binary name, same commands, one npx in front:

npx deepeval inspect is the one I'd try first. Traces are written to a local .json file on your machine by default — nothing leaves your laptop — and inspect renders them as a trace tree with per-span scores and metric reasons. When a coding agent is driving the loop, that's what stops it from overfitting to a number it can't see the reasoning behind.

What's missing, and why it's a beta

I'd rather tell you than let you find out. Three things exist in Python and don't exist in TypeScript yet:

• Synthesizer — generating goldens from your documents or knowledge base. Bring your own dataset for now: load goldens from a CSV, a JSON file, or Confident AI.
• Benchmarks — MMLU, HellaSwag, and the rest of the foundational-model benchmark suite.
• Prompt optimization — automatic prompt search against a metric.

There's also no score-parity guarantee. The prompts are shared and parity-checked, but we've spot-verified numeric scores for sanity rather than asserted them equal to Python's. Don't mix languages inside one longitudinal comparison yet.

That's what the beta label is for. Everything above it — metrics, models, tracing, the CI/CD gate, the CLI — is what we're asking you to actually use and break.

Getting started

npm install -D deepeval
npx deepeval set-openai --model gpt-4.1
npx deepeval test run llm_app.test.ts

Or don't write the test file yourself. Install the DeepEval skill and let your coding agent drive the eval driven development loop:

npx skills add confident-ai/deepeval --skill "deepeval"

Python still leads on behavior and TypeScript follows close behind — that hasn't changed, and one repo is what keeps "close behind" true. What changed is that "close behind" now means 47 metrics, 13 model providers, 6 tracing integrations, and a command that turns red on your pull requests, rather than a client that couldn't score anything.

DeepEval is free and 100% open-source on ⭐ GitHub. If TypeScript is your stack, this is the release I've been wanting to write for a year — go break it and open an issue.

FAQs