Introduction to DeepEval

DeepEval is an open-source LLM evaluation framework for LLM applications. DeepEval makes it extremely easy to build and iterate on LLM (applications) and was built with the following principles in mind:

• Unit test LLM outputs with Pytest-style assertions.
• Use 50+ ready-to-use metrics, including LLM-as-a-judge, agent, tool-use, conversational, safety, RAG, and multimodal metrics.
• Evaluate AI agents, conversational agents (chatbots), RAG pipelines, MCP systems, and other custom workflows.
• Run both end-to-end evals and component-level evals with tracing.
• Generate synthetic datasets for edge cases that are hard to collect manually.
• Customize metrics, prompts, models, and evaluation templates when built-in behavior is not enough.

DeepEval is local-first: your evaluations run in your own environment. When your team needs shared dashboards, regression tracking, observability, or production monitoring, DeepEval integrates natively with Confident AI.

Who is DeepEval For?

DeepEval was designed for a technical audience and here are the main personas we serve well:

• AI engineers who need to evaluate agents, RAG pipelines, tool calls, and production LLM workflows, write unit tests for AI behavior, and use evals in agentic coding tools like Claude Code and Codex.
• Data scientists who want repeatable experiments for comparing prompts, models, datasets, and metric scores.
• QAs who need reliable regression tests for AI behavior before changes reach users.
• Tech-savvy PMs who want to define quality criteria, inspect failures, and track whether product changes improve AI outputs.

Using DeepEval for Coding Agents

Apart from building evaluation suites and pipelines with DeepEval, DeepEval's CLI evaluation capabilities make it one of the best eval harnesses for vibe coding agents such as Claude Code, Codex, and Cursor.

The diagram below explains how DeepEval can take part in your iteration cycles, not just as a final validation check.

Choose Your Path

We highly recommend starting with either of these two quickstarts:

Start with a Use Case in Mind

Alternatively, if you already have a concrete use case - try out one of our use case specific quickstarts:

AI Agents

Set up tracing, evaluate end-to-end task completion, and score individual agent components.

Chatbots

Evaluate multi-turn conversations, turns, and simulated user interactions.

RAG

Evaluate RAG quality end-to-end, then test retrieval and generation separately.

More Resources

The Core Building Blocks

These concepts show up throughout DeepEval and learning these fundamentals are imperative:

Two Modes of Evals

DeepEval supports two complementary ways to evaluate your application, it's important to know which one(s) suit you:

You can use either mode independently, or combine them: score the whole trace for overall task quality, then score individual spans to find where failures happen.

DeepEval Ecosystem

DeepEval can run by itself, but it also connects to adjacent tools when your workflow needs collaboration, monitoring, or security testing.

Quick Shoutout To Our Community

DeepEval is shaped by the people who report bugs, propose ideas, review changes, improve docs, and ship code with us. Thank you for building this project with us.

penguine-ip — 4,314 contributionsA-Vamshi — 1,164 contributionsjwongster2 — 990 contributionskritinv — 907 contributionsspike-spiegel-21 — 732 contributionsBloggerBust — 389 contributionstrevor-cai — 90 contributionsAnindyadeep — 66 contributionstanayvaswani — 53 contributionsVasilije1990 — 28 contributionsPratyush-exe — 24 contributionsSidhaarth-Murali — 20 contributionsjohn-lemmon-lime — 18 contributionsagokrani — 17 contributionsSai-Suraj-27 — 15 contributionsfetz236 — 14 contributionsPeilun-Li — 13 contributionsvjsliogeris — 12 contributionsluarss — 11 contributionslesar64 — 10 contributionsfschuh — 9 contributionsAndrea23Romano — 7 contributionsj-space-b — 7 contributionssergeyklay — 7 contributionsAbhishekRP2002 — 6 contributionsChristianBernhard — 6 contributionskarankulshrestha — 6 contributionswjunwei2001 — 6 contributionsadityabharadwaj198 — 5 contributionsAlexMaggioni — 5 contributionsntgussoni — 5 contributionsps2program — 5 contributionsramipellumbi — 5 contributionsseankelley-dt — 5 contributionsshippy — 5 contributionsyalishanda42 — 5 contributionsAadamHaq — 4 contributionsAahilShaikh — 4 contributionsaerosta — 4 contributionsAisha630 — 4 contributionsAndresPrez — 4 contributionsBjarniHaukur — 4 contributionsbrian-romain — 4 contributionscallmephilip — 4 contributionsdaehuikim — 4 contributionsfabian57fabian — 4 contributionsjoaopmatias — 4 contributionspaul91 — 4 contributionsreal-jiakai — 4 contributionsSamSi0322 — 4 contributionsStu-ops — 4 contributionsSYED-M-HUSSAIN — 4 contributionstharun634 — 4 contributionstrevor-inflection — 4 contributionsumuthopeyildirim — 4 contributionsYleisnero — 4 contributionsyzhao244 — 4 contributionsaandyw — 3 contributionsAnanyaRaval — 3 contributionsbofenghuang — 3 contributionsbostadynamics — 3 contributionsBr1an67 — 3 contributionschuqingG — 3 contributionselsatch — 3 contributionsFizza-Mukhtar — 3 contributionshannex — 3 contributionsjoaopbini — 3 contributionsMartinoMensio — 3 contributionsobadakhalili — 3 contributionsOluwa-nifemi — 3 contributionspedroallenrevez — 3 contributionsphungpx — 3 contributionsppon1086 — 3 contributionssiesto1elemento — 3 contributionsSpectavi — 3 contributionstbeadle — 3 contributionsvandenn — 3 contributionsandres-ito-traversal — 2 contributionsAngelenx — 2 contributionsAnush008 — 2 contributionsbderenzi — 2 contributionsCAW-nz — 2 contributionschododom — 2 contributionsdanerlt — 2 contributionsdermodmaster — 2 contributionsdhinkris — 2 contributionsdonaldwasserman — 2 contributionsdunnkers — 2 contributionskbarendrecht — 2 contributionskhannurien — 2 contributionskinga-marszalkowska — 2 contributionskonerzajakub — 2 contributionskrishna0125 — 2 contributionslouisbrulenaudet — 2 contributionsLucasLeRay — 2 contributionslwarsaame — 2 contributionsmarr75 — 2 contributionsmdsalnikov — 2 contributionsmikkeyboi — 2 contributionsnabeel-chhatri — 2 contributionsNikyParfenov — 2 contributionsoftenfrequent — 2 contributionsPradyMagal — 2 contributionsraphaeluzan — 2 contributionsrohinish404 — 2 contributionsRussell-Day — 2 contributionsS3lc0uth — 2 contributionssisp — 2 contributionssobs0 — 2 contributionstiffanychum — 2 contributionsyudhiesh — 2 contributionsyujiiroo — 2 contributions88roy88 — 1 contributiona-romero — 1 contributionAaryanverma — 1 contributionacompa — 1 contributionadityamehra — 1 contributionagent-kira — 1 contributionAjay6601 — 1 contributionAmaliMatharaarachchi — 1 contributionAMindToThink — 1 contributionamrakshay — 1 contributionAugmentMo — 1 contributionbmerkle — 1 contributionbongho — 1 contributionbowenliang123 — 1 contributioncancelself — 1 contributioncastelo-software — 1 contribution+134

FAQs