💥 BREAKING CHANGE: All metric scores are now HIGHER THE BETTER. Read changelog →
Orchestration Frameworks

LangChain

Native Instrumentation
Evals in CI/CD
Evals with Traceability

LangChain is an open-source framework for building LLM applications with models, prompts, tools, retrievers, and agents (via create_agent).

The deepeval integration traces LangChain runs through a CallbackHandler that you pass into LangChain's config. Every agent run, model call, tool call, and retriever call becomes a span you can inspect, without rewriting your LangChain app.

deepeval's LangChain integration enables you to:

  • Trace any LangChain run — pass a CallbackHandler through LangChain's callbacks config per call.
  • Evaluate the complete ordered agent trajectory — score the sequence across all applicable agent, LLM, tool, and retriever spans.
  • Evaluate traces or individual components with deepeval metrics.
  • Run evals from scripts or CI/CD — same callback, different surfaces.
  • Customize trace and span data through callback kwargs and next_*_span staging.

Getting Started

Installation

pip install -U deepeval langchain langchain-openai

LangChain is instrumented per-call: you decide which runs are traced by passing CallbackHandler(...) into LangChain's runtime config.

Instrument and evaluate

Create a CallbackHandler and pass it to the agent's invoke method.

langchain_agent.py
from langchain.agents import create_agent
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

def multiply(a: int, b: int) -> int:
    """Multiply two numbers."""
    return a * b

agent = create_agent(
    model="openai:gpt-4o-mini",
    tools=[multiply],
    system_prompt="Be concise.",
)

# Goldens are the inputs you want to evaluate.
dataset = EvaluationDataset(goldens=[Golden(input="What is 8 multiplied by 6?")])

# The `TaskCompletionMetric` is passed into the `evals_iterator`.
for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    agent.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )

Done ✅. You've run your first eval with full traceability into LangChain via deepeval.

What gets traced

Each LangChain call that receives a CallbackHandler produces a trace — the end-to-end unit your user observes. Inside that trace are component spans for each callback LangChain emits:

  • Agent span — the top-level create_agent / runnable invoke(...) call (the root of the tree).
  • LLM spans — chat model and completion calls.
  • Tool spans — tool calls and function executions.
  • Retriever spans — retriever calls, when your app uses retrieval.
Trace                           ← what the user observes
└── Agent: math_agent            ← one create_agent invoke(...) call
    ├── LLM: gpt-4o-mini        ← component span: model chooses a tool
    ├── Tool: multiply          ← component span: tool input + output
    └── LLM: gpt-4o-mini        ← component span: final answer

The trace and its component spans are independently evaluable.

Running evals

There are two surfaces for running evals against a LangChain app. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.

In CI/CD

Use the deepeval pytest integration. Each parametrized test invocation becomes one LangChain run; failing metrics fail the test, which fails the build.

test_langchain_agent.py
import pytest
from langchain.agents import create_agent
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric
from deepeval import assert_test

def multiply(a: int, b: int) -> int:
    """Multiply two numbers."""
    return a * b

agent = create_agent(model="openai:gpt-4o-mini", tools=[multiply], system_prompt="Be concise.")
dataset = EvaluationDataset(goldens=[
    Golden(input="What is 8 multiplied by 6?"),
    Golden(input="What is 7 multiplied by 9?"),
])

@pytest.mark.parametrize("golden", dataset.goldens)
def test_langchain_agent(golden: Golden):
    agent.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )
    assert_test(golden=golden, metrics=[TaskCompletionMetric()])

Run it with:

deepeval test run test_langchain_agent.py

In a script

Use EvaluationDataset + evals_iterator. Each Golden becomes one LangChain run, and metrics passed to the iterator score the resulting trace end-to-end.

langchain_agent.py
dataset = EvaluationDataset(goldens=[
    Golden(input="What is 8 multiplied by 6?"),
    Golden(input="What is 7 multiplied by 9?"),
])

for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    agent.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )

Evaluate agent trajectories

Trajectory evaluation scores the complete ordered trace: the agent's decisions, model calls, tool use, and other applicable spans in the sequence they occurred. This is distinct from evaluating only the final output or attaching a metric to an individual component span.

Pass trajectory metrics to evals_iterator so each instrumented LangChain run is evaluated at trace scope:

langchain_agent.py
from deepeval.metrics import TaskCompletionMetric, StepEfficiencyMetric, PlanAdherenceMetric
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
...

dataset = EvaluationDataset(goldens=[Golden(input="What is 8 multiplied by 6?")])
metrics = [TaskCompletionMetric(), StepEfficiencyMetric(), PlanAdherenceMetric()]

for golden in dataset.evals_iterator(metrics=metrics):
    agent.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )

See the trajectory evaluation guide for metric selection and interpretation. To score one step instead of the ordered run, apply metrics to components next.

Applying metrics to components

Passing metrics to evals_iterator evaluates the overall LangChain run. To evaluate a component instead, stage metrics onto the next span the callback opens.

Agent spans

Wrap the invocation in next_agent_span. The CallbackHandler drains the staged metric onto the root agent span opened by invoke(...) — useful when you want a span-level score on the agent itself rather than the whole trace.

langchain_agent.py
from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import next_agent_span
...

for golden in dataset.evals_iterator():
    with next_agent_span(metrics=[TaskCompletionMetric()]):
        agent.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )

LLM calls

Wrap the invocation in next_llm_span. The CallbackHandler drains the staged metric onto the first LLM span it opens inside the block; later LLM calls in the same run get nothing.

langchain_agent.py
from langchain.agents import create_agent
from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import next_llm_span
...

agent = create_agent(model="openai:gpt-4o-mini", tools=[multiply], system_prompt="Be concise.")

for golden in dataset.evals_iterator():
    with next_llm_span(metrics=[AnswerRelevancyMetric()]):
        agent.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )

Retriever calls

Wrap the invocation in next_retriever_span to stage a metric (or a Confident AI metric_collection) on the first retriever span LangChain opens inside the block.

langchain_agent.py
from deepeval.integrations.langchain import CallbackHandler
from deepeval.tracing import next_retriever_span
...

for golden in dataset.evals_iterator():
    with next_retriever_span(metric_collection="retriever_v1"):
        chain.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )

Tool calls

For deterministic tool calls, use tool spans for traceability, inputs, outputs, and metadata. Prefer attaching metrics at the LLM or trace level rather than on tools.

Use deepeval's patched tool decorator when you want to attach a metric collection (or metrics) from the tool definition:

langchain_agent.py
from deepeval.integrations.langchain import tool

@tool(metric_collection="tools_v1")
def multiply(a: int, b: int) -> int:
    """Multiply two numbers."""
    return a * b

Customizing trace and span data

LangChain is instrumented per-call through callbacks, so customization happens at the callback or span-staging boundary.

  • Use CallbackHandler kwargs for trace-level defaults like name, tags, metadata, thread_id, and user_id.
  • Use next_agent_span / next_llm_span / next_retriever_span / next_tool_span to stage component-level fields onto the next span the callback opens.
  • Use tool bodies (or Python's patched tool decorator) for deterministic traceability, inputs, outputs, and metadata.
langchain_agent.py
callback = CallbackHandler(
    name="math-agent",
    tags=["langchain", "math"],
    metadata={"team": "support"},
    user_id="user-123",
)

agent.invoke(
    {"messages": [{"role": "user", "content": "What is 8 multiplied by 6?"}]},
    config={"callbacks": [callback]},
)

Advanced patterns

The primitives above — CallbackHandler and next_*_span — compose around one boundary: LangChain owns the callback lifecycle, and your code chooses where to stage component config for the next span the callback opens.

Score every matching span

Where next_*_span is one-shot, a scope-wide context applies to every matching span. Use this when an agent makes several model calls per run and you want all of them scored.

langchain_agent.py
from deepeval.tracing import LlmSpanContext, trace
from deepeval.metrics import AnswerRelevancyMetric
...

with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
    agent.invoke(
        {"messages": [{"role": "user", "content": prompt}]},
        config={"callbacks": [CallbackHandler()]},
    )

Evaluate components without trace-level metrics

Stage a metric on an LLM (or retriever) span, then run CI/CD or scripts without also passing metrics to the iterator / matcher. Trace-level metrics are end-to-end metrics: they are not strictly necessary when the component already carries the metric.

langchain_agent.py
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import next_llm_span
...

def run_agent(prompt: str):
    with next_llm_span(metrics=[AnswerRelevancyMetric()]):
        return agent.invoke(
            {"messages": [{"role": "user", "content": prompt}]},
            config={"callbacks": [CallbackHandler()]},
        )

No trace-level metrics required

This is how you'd run it:

test_langchain_agent.py
import pytest
from deepeval import assert_test
...

@pytest.mark.parametrize("golden", dataset.goldens)
def test_component_metrics(golden: Golden):
    run_agent(golden.input)
    assert_test(golden=golden)
deepeval test run test_langchain_agent.py
langchain_agent.py
...

for golden in dataset.evals_iterator():
    run_agent(golden.input)

Wrap a LangChain run in @observe

When the LangChain call is part of a larger operation, decorate the outer function with @observe. LangChain spans nest under your observed span when the callback runs inside it.

langchain_agent.py
from deepeval.tracing import observe
...

@observe(name="respond_to_user")
def respond_to_user(prompt: str) -> str:
    result = agent.invoke(
        {"messages": [{"role": "user", "content": prompt}]},
        config={"callbacks": [CallbackHandler()]},
    )
    return result["messages"][-1].content

API reference

CallbackHandler accepts the following trace-level kwargs. Each one is a default for runs that use that callback.

KwargTypeDescription
namestrDefault trace name.
tagslist[str]Tags applied to traces produced by this callback.
metadatadictTrace metadata applied when the callback starts a trace.
thread_idstrGroups related runs into a single trace thread.
user_idstrActor identifier for the trace.
metricslistTrace-level metrics for the run. Prefer evals_iterator in eval scripts.
metric_collectionstrTrace-level metric collection, for online evals on live traffic.
test_case_idstrOptional test case identifier.
turn_idstrOptional turn identifier for conversational traces.

For native tracing helpers (@observe, update_current_trace, update_current_span) see the tracing reference.

FAQs

Can I evaluate a component inside my LangChain agent run?
Yes. Stage a metric with next_agent_span / next_llm_span (or the retriever / tool helpers) around agent.invoke(...). The CallbackHandler drains it onto the first matching span. It is one-shot per run, so to score every step use a scope-wide context (trace(...)) or trace-level metrics on evals_iterator.
Can I gate CI/CD on my LangChain agent's metrics?
Yes. Pass a CallbackHandler() into the agent's config inside a parametrized pytest test, then call assert_test(golden=golden, metrics=[...]) and run deepeval test run.
Can I see these LangChain traces in a cloud UI?
Yes, optionally. After deepeval login, Confident AI renders every agent, LLM, tool, and retriever span produced by the CallbackHandler in a shared dashboard — no code changes.
Can I monitor a LangChain app in production?
Yes. Keep the CallbackHandler in your production calls and set thread_id / user_id for grouping; when logged into Confident AI those live traces support online evals on real traffic.

Google ADK

Previous Page

LangGraph

Next Page

On this page

Version CompatibilityGetting StartedInstallationInstrument and evaluateWhat gets tracedRunning evalsIn CI/CDIn a scriptEvaluate agent trajectoriesApplying metrics to componentsAgent spansLLM callsRetriever callsTool callsCustomizing trace and span dataAdvanced patternsScore every matching spanEvaluate components without trace-level metricsNo trace-level metrics requiredNo trace-level metrics requiredWrap a LangChain run in @observeAPI referenceFAQs