LangGraph

LangGraph is a low-level orchestration framework for building stateful, graph-based agent workflows. You compose agents from StateGraph nodes and edges, with full control over routing, state, and tool execution.

The deepeval integration traces LangGraph runs through LangChain's CallbackHandler, which you pass into your graph's runtime config. Every graph run, node, model call, tool call, and nested step becomes a span you can inspect, without rewriting your LangGraph app.

deepeval's LangGraph integration enables you to:

• Trace any LangGraph run — pass a CallbackHandler through the graph's callbacks config per call.
• Evaluate the complete ordered agent trajectory — score the sequence across all applicable graph, agent, LLM, and tool spans.
• Evaluate traces or model / agent components with deepeval metrics.
• Run evals from scripts or CI/CD — same callback, different surfaces.
• Customize trace and span data through callback kwargs and next_*_span staging.

Getting Started

pip install -U deepeval langgraph langchain-openai

from langchain.chat_models import init_chat_model
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

llm = init_chat_model("openai:gpt-4o-mini").bind_tools([get_weather])

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_node("tools", ToolNode([get_weather]))
    .add_edge(START, "chatbot")
    .add_conditional_edges("chatbot", tools_condition)
    .add_edge("tools", "chatbot")
    .compile()
)

# Goldens are the inputs you want to evaluate.
dataset = EvaluationDataset(goldens=[Golden(input="What is the weather in Paris?")])

# The `TaskCompletionMetric` is passed into the `evals_iterator`.
for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )

What gets traced

Each LangGraph run that receives a CallbackHandler produces a trace — the end-to-end unit your user observes. Inside that trace are component spans for each callback LangGraph emits through LangChain:

• Agent span — the compiled StateGraph invoke(...) call (the root of the tree). Nested graph nodes stay custom spans.
• LLM spans — chat model and completion calls inside a node.
• Tool spans — tool calls executed by ToolNode (or your own).
• Retriever spans — retriever calls, when your graph uses retrieval.

Trace                           ← what the user observes
└── Agent: weather_graph         ← one graph invoke(...) call
    ├── Node: chatbot           ← model picks a tool
    │   └── LLM: gpt-4o-mini
    ├── Node: tools             ← ToolNode runs the tool
    │   └── Tool: get_weather
    └── Node: chatbot           ← model writes the final answer
        └── LLM: gpt-4o-mini

The trace and its component spans are independently evaluable.

Running evals

There are two surfaces for running evals against a LangGraph app. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.

In CI/CD

Use the deepeval pytest integration. Each parametrized test invocation becomes one LangGraph run; failing metrics fail the test, which fails the build.

import pytest
from langchain.chat_models import init_chat_model
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from deepeval import assert_test
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

llm = init_chat_model("openai:gpt-4o-mini").bind_tools([get_weather])

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_node("tools", ToolNode([get_weather]))
    .add_edge(START, "chatbot")
    .add_conditional_edges("chatbot", tools_condition)
    .add_edge("tools", "chatbot")
    .compile()
)

dataset = EvaluationDataset(goldens=[
    Golden(input="What is the weather in Paris?"),
    Golden(input="What is the weather in London?"),
])

@pytest.mark.parametrize("golden", dataset.goldens)
def test_langgraph_agent(golden: Golden):
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )
    assert_test(golden=golden, metrics=[TaskCompletionMetric()])

Run it with:

deepeval test run test_langgraph_agent.py

In a script

Use EvaluationDataset + evals_iterator. Each Golden becomes one LangGraph run, and metrics passed to the iterator score the resulting trace end-to-end.

dataset = EvaluationDataset(goldens=[
    Golden(input="What is the weather in Paris?"),
    Golden(input="What is the weather in London?"),
])

for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )

Evaluate agent trajectories

Trajectory evaluation scores the complete ordered trace: the graph's routing decisions, model calls, tool use, and other applicable spans in the sequence they occurred. This is distinct from evaluating only the final output or attaching a metric to an individual component span. Custom graph and node spans remain part of that ordered trajectory, so graph-specific orchestration is preserved in the evaluation scope.

Pass trajectory metrics to evals_iterator so each instrumented LangGraph run is evaluated at trace scope:

from deepeval.metrics import TaskCompletionMetric, StepEfficiencyMetric, PlanAdherenceMetric
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
...

dataset = EvaluationDataset(goldens=[Golden(input="What is the weather in Paris?")])
metrics = [TaskCompletionMetric(), StepEfficiencyMetric(), PlanAdherenceMetric()]

for golden in dataset.evals_iterator(metrics=metrics):
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )

See the trajectory evaluation guide for metric selection and interpretation. To score one step instead of the ordered run, apply metrics to components next.

Applying metrics to components

Passing metrics to evals_iterator evaluates the overall LangGraph run. To evaluate a component instead, stage metrics onto the next span the callback opens.

Agent spans

Wrap the graph invocation in next_agent_span. The callback drains the staged metric onto the root agent span opened by invoke(...) — useful when you want a span-level score on the graph run itself rather than the whole trace.

from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import next_agent_span
...

for golden in dataset.evals_iterator():
    with next_agent_span(metrics=[TaskCompletionMetric()]):
        graph.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )

The same one-shot semantic as the LLM helper applies: only the root agent span picks up the staged metric.

LLM calls

Wrap the graph invocation in next_llm_span. The callback drains the staged metric onto the first LLM span the graph emits; later LLM calls on subsequent loop turns get nothing.

from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import next_llm_span
...

for golden in dataset.evals_iterator():
    with next_llm_span(metrics=[AnswerRelevancyMetric()]):
        graph.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )

For deterministic tool calls, use tool spans for traceability, inputs, outputs, and metadata. Avoid attaching metrics directly to tool spans.

Customizing trace and span data

LangGraph is instrumented per-call through LangChain callbacks, so customization happens at the callback or span-staging boundary.

• Use CallbackHandler kwargs for trace-level defaults like name, tags, metadata, thread_id, and user_id.
• Use next_agent_span / next_llm_span / next_retriever_span / next_tool_span to stage component-level fields onto the next span the callback opens.
• Use tool spans for deterministic traceability, inputs, outputs, and metadata.

callback = CallbackHandler(
    name="weather-graph",
    tags=["langgraph", "weather"],
    metadata={"team": "support"},
    user_id="user-123",
)

graph.invoke(
    {"messages": [{"role": "user", "content": "What is the weather in Paris?"}]},
    config={"callbacks": [callback]},
)

Advanced patterns

The primitives above — the callback handler and next_*_span — compose around one boundary: LangGraph owns the graph execution lifecycle, and your code chooses where to stage component config for the next span the callback opens.

Evaluate a node instead of the whole graph

Staging a metric applies it to the next matching span the callback opens during the graph run. Use this when you want to evaluate a subagent node or model step instead of the full graph. Pick the helper that matches the span you want to score.

from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import next_agent_span
...

def run_graph(prompt: str):
    with next_agent_span(metrics=[TaskCompletionMetric()]):
        return graph.invoke(
            {"messages": [{"role": "user", "content": prompt}]},
            config={"callbacks": [CallbackHandler()]},
        )

No trace-level metrics required

Trace-level metrics are end-to-end metrics: they score the whole trace. They are not strictly necessary here because the TaskCompletionMetric is staged for the agent span, so CI/CD and scripts only need to run the graph inside the staging block.

This is how you'd run it:

import pytest
from deepeval import assert_test
...

@pytest.mark.parametrize("golden", dataset.goldens)
def test_component_metrics(golden: Golden):
    run_graph(golden.input)
    assert_test(golden=golden)

deepeval test run test_langgraph_agent.py

...

for golden in dataset.evals_iterator():
    run_graph(golden.input)

Wrap a LangGraph run in an observed function

When the LangGraph call is part of a larger operation, observe the outer function. LangGraph spans nest under your observed span when the callback runs inside it.

from deepeval.tracing import observe
...

@observe(name="respond_to_user")
def respond_to_user(prompt: str):
    return graph.invoke(
        {"messages": [{"role": "user", "content": prompt}]},
        config={"callbacks": [CallbackHandler()]},
    )

Tracing a LangGraph server deployment

When you deploy with the LangGraph server, the server invokes your graph. So instead you can attach the handler to the compiled graph and export that. The callback then applies to every run the server executes.

from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from deepeval.integrations.langchain import CallbackHandler
...

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_node("tools", ToolNode([get_weather]))
    .add_edge(START, "chatbot")
    .add_conditional_edges("chatbot", tools_condition)
    .add_edge("tools", "chatbot")
    .compile()
    .with_config(callbacks=[CallbackHandler()])  # baked into every server run
)

Point langgraph.json at the exported graph:

{
  "dependencies": ["."],
  "graphs": { "agent": "./langgraph_agent.py:graph" },
  "env": ".env"
}

Then start the server — every request it runs against the graph is traced:

pip install -U "langgraph-cli[inmem]"
langgraph dev

API reference

CallbackHandler accepts the following trace-level kwargs. Each one is a default for runs that use that callback.

For native tracing helpers (@observe, update_current_trace, update_current_span) see the tracing reference.

FAQs