End-to-End LLM Evaluation

End-to-end evaluation assesses the observable inputs and outputs of your LLM application and treats it as a black box — you only care about what goes in and what comes out, not the path the system took to get there. The shape of "input" and "output" depends entirely on what your app does:

• Tool-using agent treated as a black box — input is the user's task, output is the final answer plus the tools that were called.
• Multi-turn chatbot / support agent — input is the scenario the user is in, output is the full conversation.
• RAG / QA app — input is a question, output is the answer (and the retrieved context, if you want to score faithfulness).
• Document summarization — input is the source document, output is the summary.
• Classifier / extractor — input is a chunk of text, output is the label or the structured fields you pulled out.
• Writing assistant / rewriter — input is the draft (and any instructions), output is the rewritten text.

This page explains the concepts behind end-to-end evaluation. For the actual step-by-step walkthroughs, jump to the right flavor for your application:

• Single-Turn End-to-End Evals — for any LLM app where one input maps to one output (agents treated as a black box, RAG / QA, summarization, classifiers, etc.).
• Multi-Turn End-to-End Evals — for chatbots and conversational agents where the unit of evaluation is the whole conversation.

If you need to evaluate the internal path taken by an AI agent or long-horizon agent, use trajectory-based evaluation instead.

Treating Your App as a Black Box

In end-to-end evaluation, you only describe what's observable from outside your LLM application — the input you sent, the output that came back, and any context that was used along the way. You do not describe the retrieval algorithm, the chain of LLM calls inside an agent, or any internal reasoning steps. That's the whole point of "end-to-end": you're grading the result, not the path the system took to get there.

Concretely, the parameters you populate on a test case are the entire surface your metrics see.

For single-turn apps, you populate fields on an LLMTestCase:

• input — what you sent into your app (the question, document, draft, task, etc.).
• actual_output — what your app produced (the answer, summary, label, rewritten text, agent's final reply).
• retrieval_context — for RAG-style apps, the chunks your retriever returned. Required by metrics like FaithfulnessMetric and ContextualRelevancyMetric.
• tools_called — for agentic apps, the tools the agent invoked. Required by metrics like ToolCorrectnessMetric and ArgumentCorrectnessMetric.
• expected_output / expected_tools — optional gold references, used by reference-based metrics.
• context — optional extra background, used by some reference-based metrics.

For multi-turn apps, you populate fields on a ConversationalTestCase:

• scenario — what the simulated user is trying to do.
• expected_outcome — what success looks like.
• turns — the sequence of Turn objects (each with a role and content) that make up the conversation.

Notice what's not there: there's no place to describe "the retriever's prompt", "the tool argument schema", or "the inner LLM call that produced this answer." If a metric needs to score one of those things in isolation, end-to-end isn't the right fit.

Single-Turn vs Multi-Turn

Pick the flavor that matches your application:

The two flavors live on different test case classes because the unit of evaluation is genuinely different (one exchange vs many), and deepeval will refuse to mix them in the same test run.

End-to-End vs Trajectory-Based vs Component-Level

End-to-end, trajectory-based, and component-level evaluation use different scopes. End-to-end sees the system as a black box, trajectory-based sees the complete chain inside an agent, and component-level isolates one internal span.

In each case, you attach metrics to a different unit of work:

• End-to-end — the unit is the whole app. One test case per run of your app, scoring the final input → final output.
• Trajectory-based — the unit is the agent's complete ordered execution path, scoring how its internal steps work together.
• Component-level — the unit is each @observe'd span. Many test cases per run of your app — one per span you've chosen to grade — each scoring the input → output of that span.

You don't have to choose just one. With the recommended evals iterator path, black-box metrics can score the final result, trajectory metrics can score the complete agent path, and metrics attached to individual spans with @observe(metrics=[...]) can score components in the same traced run.

When should you choose end-to-end?

Choose end-to-end evaluation when:

• Your LLM application has a "flat" architecture that fits naturally into a single LLMTestCase (agents treated as a black box, RAG / QA, summarization, single-shot classifiers, writing assistants, etc.)
• Your application is multi-turn (chatbots, support agents) and you want to score the whole conversation rather than each step.
• Your application is a complex agent, but you've concluded that component-level evaluation gives you too much noise and you'd rather grade the final outcome.

In short: you care about the result, not the path the system took to get there. Most of the quickstart is end-to-end evaluation.

Two Ways to Run a Test Run

Both single-turn and (for evaluate()) multi-turn give you a choice between two equivalent code paths:

For new single-turn projects we recommend the iterator — same amount of code, plus traces, plus the same setup carries over to component-level evaluation later.

Multi-turn end-to-end evaluation only uses evaluate() today; the iterator form is single-turn only.

What's Next

• Walk through a single-turn end-to-end evaluation.
• Walk through a multi-turn end-to-end evaluation using the ConversationSimulator.
• Evaluate the complete internal path of an agent with trajectory-based evaluation.
• Run end-to-end evals in CI/CD pipelines using pytest and deepeval test run.
• Compare with component-level evaluation if your app has internal structure worth grading.