Vercel AI SDK

Vercel AI SDK is a TypeScript toolkit for calling LLMs — generateText, streamText, tools, structured output, and embeddings — with built-in OpenTelemetry spans for every step.

The deepeval integration registers an OpenTelemetry processor that turns those ai.* spans into traces you can inspect and evaluate. No rewrite of your AI SDK calls beyond enabling telemetry.

deepeval's Vercel AI SDK integration enables you to:

• Trace every AI SDK call — configureAiSdkTracing(...) once, then pass the tracer into experimental_telemetry on each call.
• Evaluate traces or individual components with any deepeval metric.
• Run evals from scripts or CI/CD — same instrumentation, different surfaces.
• Customize trace and span data through configure options, AI SDK telemetry metadata, and deepeval's span-staging helpers.

Getting Started

npm install deepeval ai @ai-sdk/openai

import { generateText, tool, stepCountIs } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";

import { configureAiSdkTracing } from "deepeval/integrations/ai-sdk";
import { EvaluationDataset, Golden } from "deepeval/dataset";
import { TaskCompletionMetric } from "deepeval/metrics";

const tracer = configureAiSdkTracing({ name: "weather-app" });

const ask = (question: string) =>
  generateText({
    model: openai("gpt-4o-mini"),
    prompt: question,
    tools: {
      getWeather: tool({
        description: "Get the current weather for a city.",
        inputSchema: z.object({ city: z.string() }),
        execute: async ({ city }) => ({
          weather: city.toLowerCase() === "tokyo" ? "Sunny, 72F" : "Rainy, 55F",
        }),
      }),
    },
    stopWhen: stepCountIs(2),
    experimental_telemetry: {
      isEnabled: true,
      tracer,
    },
  });

// Goldens are the inputs you want to evaluate.
const dataset = new EvaluationDataset({
  goldens: [new Golden({ input: "What's the weather in Tokyo?" })],
});

// The `TaskCompletionMetric` is passed into the `evalsIterator`.
for await (const golden of dataset.evalsIterator({
  metrics: [new TaskCompletionMetric()],
})) {
  await ask((golden as Golden).input);
}

What gets traced

Each top-level AI SDK call (generateText, streamText, generateObject, streamObject, embed, embedMany) that has telemetry enabled produces a trace. Inside it, Vercel’s ai.* spans map to deepeval span types:

Trace                              ← what the user observes
└── LLM: ai.generateText            ← one generateText(...) call
    ├── LLM: ai.generateText.doGenerate   ← model chooses a tool
    ├── Tool: getWeather                  ← tool input + output
    └── LLM: ai.generateText.doGenerate   ← final answer

The trace and its component spans are independently evaluable. Every tool call is also collected onto the trace's toolsCalled so trace-level metrics can reason about tool use.

There is no agent span — AI SDK's root is the generation call itself. Stage LLM / tool / retriever metrics with nextLlmSpan / nextToolSpan / nextRetrieverSpan, or score the whole run with trace-level metrics on evalsIterator.

Running evals

There are two surfaces for running evals against an AI SDK app. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.

In CI/CD (Vitest)

Use the toPass() matcher. The golden is the subject; task produces the trace judged against it. A failing metric fails the test, which fails the build.

import { it, expect } from "vitest";
import { EvaluationDataset, Golden } from "deepeval/dataset";
import { TaskCompletionMetric } from "deepeval/metrics";
import { ask } from "./ai-sdk-agent";
import "deepeval/vitest";

const dataset = new EvaluationDataset({
  goldens: [
    new Golden({ input: "What's the weather in Tokyo?" }),
    new Golden({ input: "What's the weather in London?" }),
  ],
});

it.each(dataset.goldens as Golden[])(
  "completes the weather task",
  async (golden) => {
    await expect(golden).toPass([new TaskCompletionMetric()], {
      task: (g) => ask(g.input),
    });
  },
);

Run it with:

npx deepeval test run ai-sdk-agent.test.ts

In a script

Use EvaluationDataset + evalsIterator(...). Each Golden becomes one AI SDK run, and metrics passed to the iterator score the resulting trace end-to-end.

const dataset = new EvaluationDataset({
  goldens: [
    new Golden({ input: "What's the weather in Tokyo?" }),
    new Golden({ input: "What's the weather in London?" }),
  ],
});

for await (const golden of dataset.evalsIterator({
  metrics: [new TaskCompletionMetric()],
})) {
  await ask((golden as Golden).input);
}

Applying metrics to components

Passing metrics=[...] to evalsIterator evaluates the overall AI SDK run. To evaluate a component instead, stage metrics onto the next span the processor opens.

LLM calls

Wrap the call in nextLlmSpan(...). The processor drains the staged metric onto the first LLM span it opens inside the callback — typically the root ai.generateText / ai.streamText span.

import { AnswerRelevancyMetric } from "deepeval/metrics";
import { nextLlmSpan } from "deepeval/tracing";
...

for await (const golden of dataset.evalsIterator()) {
  await nextLlmSpan({ metrics: [new AnswerRelevancyMetric()] }, () =>
    ask((golden as Golden).input),
  );
}

Tool calls

Inside a tool execute body your code is the tool span, so reach it with updateCurrentSpan(...):

import { updateCurrentSpan } from "deepeval/tracing";
import { tool } from "ai";
import { z } from "zod";
...

const getWeather = tool({
  description: "Get the current weather for a city.",
  inputSchema: z.object({ city: z.string() }),
  execute: async ({ city }) => {
    updateCurrentSpan({
      metadata: { operation: "getWeather" },
    });
    return {
      weather: city.toLowerCase() === "tokyo" ? "Sunny, 72F" : "Rainy, 55F",
    };
  },
});

nextToolSpan(...) stages the same fields from outside the tool, on the same one-shot semantic as nextLlmSpan.

Retriever calls (embeddings)

ai.embed / ai.embedMany become retriever spans, so nextRetrieverSpan(...) stages a metric (or a Confident AI metricCollection) on the first one in the callback.

import { embed } from "ai";
import { openai } from "@ai-sdk/openai";
import { nextRetrieverSpan } from "deepeval/tracing";
...

for await (const golden of dataset.evalsIterator()) {
  await nextRetrieverSpan({ metricCollection: "embeddings_v1" }, () =>
    embed({
      model: openai.embedding("text-embedding-3-small"),
      value: (golden as Golden).input,
      experimental_telemetry: { isEnabled: true, tracer },
    }),
  );
}

Customizing trace and span data

AI SDK owns the span lifecycle, so customization happens at configure time, per-call telemetry metadata, or the span-staging boundary.

• Use configureAiSdkTracing({...}) for defaults that apply to every trace.
• Use experimental_telemetry.metadata / functionId for values that change per call.
• Use nextLlmSpan(...) / nextToolSpan(...) / nextRetrieverSpan(...) to stage component-level fields onto the next span the processor opens.
• Use updateCurrentSpan(...) inside a tool body, where your code already sits on the span.

Configure-level defaults:

const tracer = configureAiSdkTracing({
  name: "weather-app",
  environment: "development",
  traceMetricCollection: "online_weather_v1",
});

Per-call telemetry — functionId becomes the trace name; recognized metadata keys become trace fields:

await generateText({
  model: openai("gpt-4o-mini"),
  prompt: "What's the weather in Tokyo?",
  experimental_telemetry: {
    isEnabled: true,
    tracer,
    functionId: "weather-ask",
    metadata: {
      threadId: "thread-123",
      userId: "user-xyz",
      tags: ["production"],
      traceName: "weather-ask",
    },
  },
});

Recognized metadata keys: threadId, userId, testCaseId, turnId, tags, context, traceName, traceMetricCollection, metricCollection, expectedOutput, sessionId, promptAlias, promptCommitHash. Other keys fall through to span metadata.

Advanced patterns

Score every matching span with setTracingContext

Where next*Span(...) is one-shot, setTracingContext(...) is scope-wide: it applies to every matching span in the callback. Use this when a tool loop makes several model calls and you want all of them scored.

import { AnswerRelevancyMetric, ToolCorrectnessMetric } from "deepeval/metrics";
import { setTracingContext } from "deepeval/tracing";
...

await setTracingContext(
  {
    llmSpanContext: {
      metrics: [new AnswerRelevancyMetric()],
      toolsMetrics: [new ToolCorrectnessMetric()],
    },
  },
  () => ask("What's the weather in Tokyo?"),
);

toolsMetrics and toolsMetricCollection are declared on llmSpanContext but target the tool spans in scope.

Streaming

streamText / streamObject produce the same span types as their non-streaming counterparts. Consume the stream fully so the span can end with a complete output:

import { streamText } from "ai";
...

const { textStream } = streamText({
  model: openai("gpt-4o-mini"),
  prompt: "Count from 1 to 5.",
  experimental_telemetry: { isEnabled: true, tracer },
});

for await (const chunk of textStream) {
  // consume chunks so the span completes
}

API reference

configureAiSdkTracing({...}) accepts the following options and returns an OpenTelemetry Tracer to pass into experimental_telemetry.

Also exported:

• createDeepEvalProcessors(options?) — build the span processors without registering a global provider (bring-your-own NodeTracerProvider).
• forceFlush() — flush the registered provider (useful before a process exits outside an eval).
• DeepEvalSpanProcessor — the processor that materializes deepeval spans from ai.* OTel spans.

For native tracing helpers (observe, updateCurrentTrace, updateCurrentSpan, nextLlmSpan) see the tracing reference.

FAQs