LangGraph

LangGraph is a low-level orchestration framework for building stateful, graph-based agent workflows. You compose agents from StateGraph nodes and edges, with full control over routing, state, and tool execution.

The deepeval integration traces LangGraph runs through LangChain's CallbackHandler, which you pass into your graph's runtime config. Every graph run, node, model call, tool call, and nested step becomes a span you can inspect, without rewriting your LangGraph app.

deepeval's LangGraph integration enables you to:

• Trace any LangGraph run — pass CallbackHandler(...) through config={"callbacks": [...]} per call.
• Evaluate traces or model / agent components with deepeval metrics.
• Run evals from scripts or CI/CD — same callback, different surfaces.
• Customize trace and span data through callback kwargs and LangChain metadata.

Getting Started

pip install -U deepeval langgraph langchain-openai

from langchain.chat_models import init_chat_model
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

llm = init_chat_model("openai:gpt-4o-mini").bind_tools([get_weather])

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_node("tools", ToolNode([get_weather]))
    .add_edge(START, "chatbot")
    .add_conditional_edges("chatbot", tools_condition)
    .add_edge("tools", "chatbot")
    .compile()
)

# Goldens are the inputs you want to evaluate.
dataset = EvaluationDataset(goldens=[Golden(input="What is the weather in Paris?")])

# The `TaskCompletionMetric` is passed into the LangGraph callback.
for golden in dataset.evals_iterator():
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
    )

What gets traced

Each LangGraph run that receives a CallbackHandler produces a trace — the end-to-end unit your user observes. Inside that trace are component spans for each callback LangGraph emits through LangChain:

• Graph / node spans — the compiled StateGraph invocation and each node it dispatches to.
• LLM spans — chat model and completion calls inside a node.
• Tool spans — tool calls executed by ToolNode (or your own).
• Retriever spans — retriever calls, when your graph uses retrieval.

Trace                           ← what the user observes
└── Graph: weather_graph         ← one graph invoke(...) call
    ├── Node: chatbot           ← model picks a tool
    │   └── LLM: gpt-4o-mini
    ├── Node: tools             ← ToolNode runs the tool
    │   └── Tool: get_weather
    └── Node: chatbot           ← model writes the final answer
        └── LLM: gpt-4o-mini

The trace and its component spans are independently evaluable.

Running evals

There are two surfaces for running evals against a LangGraph app. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.

In CI/CD (pytest)

Use the deepeval pytest integration. Each parametrized test invocation becomes one LangGraph run; failing metrics fail the test, which fails the build.

import pytest
from langchain.chat_models import init_chat_model
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from deepeval import assert_test
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

llm = init_chat_model("openai:gpt-4o-mini").bind_tools([get_weather])

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_node("tools", ToolNode([get_weather]))
    .add_edge(START, "chatbot")
    .add_conditional_edges("chatbot", tools_condition)
    .add_edge("tools", "chatbot")
    .compile()
)

dataset = EvaluationDataset(goldens=[
    Golden(input="What is the weather in Paris?"),
    Golden(input="What is the weather in London?"),
])

@pytest.mark.parametrize("golden", dataset.goldens)
def test_langgraph_agent(golden: Golden):
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )
    assert_test(golden=golden, metrics=[TaskCompletionMetric()])

Run it with:

deepeval test run test_langgraph_agent.py

In a script

Use EvaluationDataset + evals_iterator(...). Each Golden becomes one LangGraph run; metrics score the resulting trace through the callback.

dataset = EvaluationDataset(goldens=[
    Golden(input="What is the weather in Paris?"),
    Golden(input="What is the weather in London?"),
])

for golden in dataset.evals_iterator():
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
    )

Applying metrics to components

Passing metrics=[...] to CallbackHandler evaluates the overall LangGraph run. To evaluate a model component instead, attach metrics where the node calls the model.

LLM calls

Wrap the graph.invoke(...) in with next_llm_span(metrics=[...]):. The CallbackHandler drains the staged metric onto the first LLM span the graph emits; later LLM calls on subsequent loop turns get nothing. This is the same one-shot semantic used by next_*_span in the Pydantic AI / Strands / AgentCore / Google ADK integrations.

from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import next_llm_span
...

for golden in dataset.evals_iterator():
    with next_llm_span(metrics=[AnswerRelevancyMetric()]):
        graph.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )

For deterministic tool calls, use tool spans for traceability, inputs, outputs, and metadata. Avoid attaching metrics directly to tool spans.

Customizing trace and span data

LangGraph is instrumented per-call through LangChain callbacks, so customization happens at the callback or span-staging boundary.

• Use CallbackHandler(...) kwargs for trace-level defaults like name, tags, metadata, thread_id, and user_id.
• Use next_llm_span(...) / next_retriever_span(...) / next_tool_span(...) to stage component-level fields (metrics, metric collections, test cases, custom span metadata) onto the next span the callback opens.
• Use tool spans for deterministic traceability, inputs, outputs, and metadata.

callback = CallbackHandler(
    name="weather-graph",
    tags=["langgraph", "weather"],
    metadata={"team": "support"},
    user_id="user-123",
)

graph.invoke(
    {"messages": [{"role": "user", "content": "What is the weather in Paris?"}]},
    config={"callbacks": [callback]},
)

Advanced patterns

The primitives above — CallbackHandler(...) and next_*_span(...) — compose around one boundary: LangGraph owns the graph execution lifecycle, and your code chooses where to stage component config for the next span the callback opens.

Evaluate subagents/components

Stage a component metric with next_llm_span(...) immediately before the graph.invoke(...) call. The CallbackHandler drains the staged metric onto the first LLM span emitted by the chatbot node, so the metric lives on the component span without modifying the graph or model.

from langchain.chat_models import init_chat_model
from langgraph.graph import StateGraph, MessagesState, START, END
from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import next_llm_span
...

llm = init_chat_model("openai:gpt-4o-mini")

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_edge(START, "chatbot")
    .add_edge("chatbot", END)
    .compile()
)

No trace-level metrics required

Trace-level metrics are end-to-end metrics: they score the whole trace. They are not strictly necessary here because the AnswerRelevancyMetric is staged for the LLM span, so CI/CD and scripts only need to run the graph inside the staging block.

This is how you'd run it:

import pytest
from deepeval import assert_test
...

@pytest.mark.parametrize("golden", dataset.goldens)
def test_component_metrics(golden: Golden):
    with next_llm_span(metrics=[AnswerRelevancyMetric()]):
        graph.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )
    assert_test(golden=golden)

deepeval test run test_langgraph_agent.py

...

for golden in dataset.evals_iterator():
    with next_llm_span(metrics=[AnswerRelevancyMetric()]):
        graph.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )

Wrap a LangGraph run in @observe

When the LangGraph call is part of a larger operation, decorate the outer function with @observe. LangGraph spans nest under your observed span when the callback runs inside it.

from deepeval.tracing import observe
...

@observe(name="respond_to_user")
def respond_to_user(prompt: str):
    return graph.invoke(
        {"messages": [{"role": "user", "content": prompt}]},
        config={"callbacks": [CallbackHandler()]},
    )

API reference

CallbackHandler(...) accepts the following trace-level kwargs. Each one is a default for runs that use that callback.

For native tracing helpers (@observe, with trace(...), update_current_trace, update_current_span) see the tracing reference.