Mastra

Mastra is a TypeScript framework for building agents, tools, and workflows, with its own observability layer that emits a span for every step an agent takes.

The deepeval integration plugs a DeepEvalExporter into that observability layer. Every agent run, model generation, tool call, and RAG operation becomes a span you can inspect and evaluate, without rewriting your Mastra app.

deepeval's Mastra integration enables you to:

• Trace every agent run — register the exporter once on your Mastra instance and every generate(...) call produces a trace.
• Evaluate traces or individual components with any deepeval metric.
• Run evals from scripts or CI/CD — same exporter, different surfaces.
• Customize trace and span data through exporter config, Mastra's per-request tracingOptions, and deepeval's span-staging helpers.

Getting Started

npm install -D deepeval @mastra/core @mastra/observability

import { Observability } from "@mastra/observability";
import { createTool } from "@mastra/core/tools";
import { Mastra } from "@mastra/core/mastra";
import { Agent } from "@mastra/core/agent";
import { z } from "zod";

import { DeepEvalExporter } from "deepeval/integrations/mastra";
import { EvaluationDataset, Golden } from "deepeval/dataset";
import { TaskCompletionMetric } from "deepeval/metrics";

const getWeather = createTool({
  id: "getWeather",
  description: "Get the current weather for a city.",
  inputSchema: z.object({ city: z.string() }),
  outputSchema: z.object({ weather: z.string() }),
  execute: async ({ city }) => ({
    weather: city.toLowerCase() === "tokyo" ? "Sunny, 72F" : "Rainy, 55F",
  }),
});

const weatherAgent = new Agent({
  id: "weather-agent",
  name: "Weather Agent",
  instructions: "Always use getWeather, then reply in one sentence.",
  model: "openai/gpt-4o-mini",
  tools: { getWeather },
});

const mastra = new Mastra({
  agents: { weatherAgent },
  observability: new Observability({
    configs: {
      deepeval: {
        serviceName: "weather-app",
        exporters: [new DeepEvalExporter()],
      },
    },
  }),
});

const ask = (question: string) =>
  mastra.getAgent("weatherAgent").generate(question);

// Goldens are the inputs you want to evaluate.
const dataset = new EvaluationDataset({
  goldens: [new Golden({ input: "What's the weather in Tokyo?" })],
});

// The `TaskCompletionMetric` is passed into the `evalsIterator`.
for await (const golden of dataset.evalsIterator({
  metrics: [new TaskCompletionMetric()],
})) {
  await ask((golden as Golden).input);
}

What gets traced

Every Mastra run produces a trace — the end-to-end unit your user observes. Inside it, each span Mastra exports is mapped to the deepeval span type that matches it:

Trace                            ← what the user observes
└── Agent: Weather Agent          ← one generate(...) call
    ├── LLM: gpt-4o-mini         ← component span: model chooses a tool
    ├── Tool: getWeather         ← component span: tool input + output
    └── LLM: gpt-4o-mini         ← component span: final answer

The trace and its component spans are independently evaluable. Streaming chunk spans (MODEL_CHUNK) and Mastra's zero-duration event spans are dropped rather than turned into noise, and every tool call is also collected onto the trace's toolsCalled so trace-level metrics can reason about tool use.

Running evals

There are two surfaces for running evals against a Mastra app. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.

In CI/CD (Vitest)

Use the toPass() matcher. The golden is the subject; task produces the trace judged against it. A failing metric fails the test, which fails the build.

import { it, expect } from "vitest";
import { EvaluationDataset, Golden } from "deepeval/dataset";
import { TaskCompletionMetric } from "deepeval/metrics";
import { ask } from "./mastra-agent";
import "deepeval/vitest";

const dataset = new EvaluationDataset({
  goldens: [
    new Golden({ input: "What's the weather in Tokyo?" }),
    new Golden({ input: "What's the weather in London?" }),
  ],
});

it.each(dataset.goldens as Golden[])(
  "completes the weather task",
  async (golden) => {
    await expect(golden).toPass([new TaskCompletionMetric()], {
      task: (g) => ask(g.input),
    });
  },
);

Run it with:

npx deepeval test run mastra-agent.test.ts

In a script

Use EvaluationDataset + evalsIterator(...). Each Golden becomes one Mastra run, and metrics passed to the iterator score the resulting trace end-to-end.

const dataset = new EvaluationDataset({
  goldens: [
    new Golden({ input: "What's the weather in Tokyo?" }),
    new Golden({ input: "What's the weather in London?" }),
  ],
});

for await (const golden of dataset.evalsIterator({
  metrics: [new TaskCompletionMetric()],
})) {
  await ask((golden as Golden).input);
}

await mastra.observability.shutdown();

Applying metrics to components

Passing metrics=[...] to evalsIterator evaluates the overall Mastra run. To evaluate a component instead, attach metrics where Mastra creates that component.

Agent spans (sub-agents)

Wrap the call in nextAgentSpan(...). The exporter drains the staged config onto the first agent span it opens inside the callback — useful for scoring a sub-agent or a workflow step in isolation.

import { TaskCompletionMetric } from "deepeval/metrics";
import { nextAgentSpan } from "deepeval/tracing";
...

for await (const golden of dataset.evalsIterator()) {
  await nextAgentSpan({ metrics: [new TaskCompletionMetric()] }, () =>
    ask((golden as Golden).input),
  );
}

LLM calls

Wrap the call in nextLlmSpan(...). The exporter drains the staged config onto the first LLM span it opens inside the callback; later model generations in the same run get nothing.

import { AnswerRelevancyMetric } from "deepeval/metrics";
import { nextLlmSpan } from "deepeval/tracing";
...

for await (const golden of dataset.evalsIterator()) {
  await nextLlmSpan({ metrics: [new AnswerRelevancyMetric()] }, () =>
    ask((golden as Golden).input),
  );
}

Staged fields also beat the exporter's static config, so a nextLlmSpan({ metricCollection }) overrides llmMetricCollection for that one span.

Tool calls

Inside a Mastra tool body, your code is the tool span, so reach it directly with updateCurrentSpan(...). This is the only place expectedTools can be supplied per call — nothing can infer it for you.

import { ToolCorrectnessMetric } from "deepeval/metrics";
import { updateCurrentSpan } from "deepeval/tracing";
import { ToolCall } from "deepeval/test-case";
...

const getWeather = createTool({
  id: "getWeather",
  description: "Get the current weather for a city.",
  inputSchema: z.object({ city: z.string() }),
  outputSchema: z.object({ weather: z.string() }),
  execute: async ({ city }) => {
    updateCurrentSpan({
      metrics: [new ToolCorrectnessMetric()],
      toolsCalled: [new ToolCall({ name: "getWeather", inputParameters: { city } })],
      expectedTools: [new ToolCall({ name: "getWeather" })],
    });

    return { weather: city.toLowerCase() === "tokyo" ? "Sunny, 72F" : "Rainy, 55F" };
  },
});

nextToolSpan(...) stages the same fields from outside the tool, on the same one-shot semantic as nextLlmSpan. Prefer it for deterministic tools where you only want traceability and metadata rather than a metric.

Retriever calls

Mastra's RAG embedding and vector-operation spans become retriever spans, so nextRetrieverSpan(...) stages a metric (or a Confident AI metricCollection) on the first one in the callback.

import { nextRetrieverSpan } from "deepeval/tracing";
...

for await (const golden of dataset.evalsIterator()) {
  await nextRetrieverSpan({ metricCollection: "retriever_v1" }, () =>
    ask((golden as Golden).input),
  );
}

Customizing trace and span data

Mastra owns the span lifecycle, so customization happens at the exporter or the span-staging boundary.

• Use new DeepEvalExporter({...}) for defaults that apply to every trace the exporter produces.
• Use Mastra's per-request tracingOptions for values that change per call — these override the exporter's defaults.
• Use nextAgentSpan(...) / nextLlmSpan(...) / nextToolSpan(...) / nextRetrieverSpan(...) to stage component-level fields onto the next span Mastra opens.
• Use updateCurrentSpan(...) inside a tool body, where your code already sits on the span.

Exporter-level defaults:

const exporter = new DeepEvalExporter({
  name: "weather-app",
  environment: "development",
  tags: ["mastra", "weather"],
  metadata: { team: "support" },
  userId: "user-123",
});

Per-request context, read off the root span's metadata and tags:

await mastra.getAgent("weatherAgent").generate("What's the weather in Tokyo?", {
  tracingOptions: {
    metadata: { threadId: "thread-123", userId: "user-xyz", team: "growth" },
    tags: ["production"],
  },
});

threadId, userId, traceName, testCaseId and turnId are read as trace fields; sessionId and resourceId are accepted as aliases for the first two. Every other key falls through to the trace's metadata, merged onto whatever the exporter's config already set.

Advanced patterns

Score every matching span with setTracingContext

Where next*Span(...) is one-shot, setTracingContext(...) is scope-wide: it applies to every matching span in the callback. This is what you want when an agent makes several model calls per run and you want all of them scored.

import { AnswerRelevancyMetric, ToolCorrectnessMetric } from "deepeval/metrics";
import { setTracingContext } from "deepeval/tracing";
...

await setTracingContext(
  {
    llmSpanContext: {
      metrics: [new AnswerRelevancyMetric()],
      toolsMetrics: [new ToolCorrectnessMetric()],
    },
  },
  () => ask("What's the weather in Tokyo?"),
);

toolsMetrics and toolsMetricCollection are declared on llmSpanContext but target the tool spans in scope. agentSpanContext does the same for agent spans.

Mastra runs own their trace

Unlike callback-based integrations, the exporter starts a new trace for each Mastra trace id rather than nesting under an enclosing span. Wrapping generate(...) in observe(...) therefore produces two sibling traces rather than one nested tree, and evalsIterator() / toPass() will warn that the callback produced more than one trace — trace-level metrics then judge only the one carrying the turn's output.

So instead of wrapping the run, put your own logic in a Mastra tool or workflow step and let Mastra parent it.

API reference

new DeepEvalExporter({...}) accepts the following config. Each one is a default for traces produced by that exporter.

The exporter never sets trace-level metrics — those come from evalsIterator({ metrics }) or toPass([...]) in an eval, and from a metricCollection in production.

For native tracing helpers (observe, updateCurrentTrace, updateCurrentSpan) see the tracing reference.

FAQs