Unit testing for LLMs.

Pytest-native evals that run in CI/CD or as Python scripts. Iterate locally, on your own environment, on your own criteria.

tests/test_agent.py

from deepeval.metrics import TaskCompletenessMetric
from deepeval.test_case import LLMTestCase
from deepeval import assert_test
 
@pytest.mark.parametrize("test_case", LLMTestCase)
def test_agent(test_case: LLMTestCase):
 my_ai_agent(test_case.input) # Captures full execution trace
 assert_test(metrics=[TaskCompletenessMetric()]) # Assert on custom criteria

scripts/run_deepeval.sh

$deepeval test run tests/test_agent.py

Evaluate

LLM-as-a-Judge to count on.

Research-backed metrics with transparent, explainable scores — every judgment comes with reasoning you can trust, debug, and defend.

Flexible, SOTA evaluation techniques.

Compose state-of-the-art techniques into metrics that fit your product — plain-English criteria, decision graphs, weighted scoring, and more, all in the same runner.

Trace, grade, and iterate — without leaving your editor.

DeepEval traces every step of your agent into something you can grade, and improve — visible in your terminal, testable in your runner, shippable in your next commit. No dashboards to open. No context switch required.

No dataset? No problem.

Generate synthetic goldens from your knowledge base, or simulate full conversations across user personas — all before a single real user shows up.

Used by agents, loved by vibe-coders.

Coming soon on May 15th...

Evaluate in code, scale with platform.

DeepEval integrates natively with Confident AI, an AI observability and evaluation platform for AI quality. It is our Vercel for DeepEval. The same test file you run on your laptop now poweres engineering, product, QAs, and domain experts.

Explore enterprise

Any model. Any framework. Any pipeline.

Plug DeepEval into the tools you already ship with — evaluate across any LLM, any agent framework, and any CI/CD runner without rewriting a line.

Built by amazing humans.

Nothing would be possible without our community of 250+ contributors, thank you!

penguine-ip — 4,193 contributionsA-Vamshi — 1,104 contributionsjwongster2 — 990 contributionskritinv — 904 contributionsspike-spiegel-21 — 732 contributionsBloggerBust — 389 contributionstrevor-cai — 90 contributionsAnindyadeep — 66 contributionstanayvaswani — 53 contributionsVasilije1990 — 28 contributionsPratyush-exe — 24 contributionsSidhaarth-Murali — 20 contributionsjohn-lemmon-lime — 18 contributionsagokrani — 17 contributionsSai-Suraj-27 — 15 contributionsfetz236 — 14 contributionsPeilun-Li — 13 contributionsvjsliogeris — 12 contributionsluarss — 11 contributionslesar64 — 10 contributionsfschuh — 9 contributionsAndrea23Romano — 7 contributionsj-space-b — 7 contributionssergeyklay — 7 contributionsAbhishekRP2002 — 6 contributionsChristianBernhard — 6 contributionskarankulshrestha — 6 contributionswjunwei2001 — 6 contributionsadityabharadwaj198 — 5 contributionsAlexMaggioni — 5 contributionsntgussoni — 5 contributionsps2program — 5 contributionsramipellumbi — 5 contributionsseankelley-dt — 5 contributionsshippy — 5 contributionsyalishanda42 — 5 contributionsAadamHaq — 4 contributionsAahilShaikh — 4 contributionsaerosta — 4 contributionsAisha630 — 4 contributionsAndresPrez — 4 contributionsBjarniHaukur — 4 contributionsbrian-romain — 4 contributionscallmephilip — 4 contributionsdaehuikim — 4 contributionsfabian57fabian — 4 contributionsjoaopmatias — 4 contributionspaul91 — 4 contributionsreal-jiakai — 4 contributionsSamSi0322 — 4 contributionsStu-ops — 4 contributionsSYED-M-HUSSAIN — 4 contributionstharun634 — 4 contributionstrevor-inflection — 4 contributionsumuthopeyildirim — 4 contributionsYleisnero — 4 contributionsaandyw — 3 contributionsAnanyaRaval — 3 contributionsbofenghuang — 3 contributionsbostadynamics — 3 contributionsBr1an67 — 3 contributionschuqingG — 3 contributionselsatch — 3 contributionsFizza-Mukhtar — 3 contributionshannex — 3 contributionsjoaopbini — 3 contributionsMartinoMensio — 3 contributionsobadakhalili — 3 contributionsOluwa-nifemi — 3 contributionspedroallenrevez — 3 contributionsphungpx — 3 contributionsppon1086 — 3 contributionssiesto1elemento — 3 contributionsSpectavi — 3 contributionstbeadle — 3 contributionsvandenn — 3 contributionsyzhao244 — 3 contributionsandres-ito-traversal — 2 contributionsAngelenx — 2 contributionsAnush008 — 2 contributionsbderenzi — 2 contributionsCAW-nz — 2 contributionschododom — 2 contributionsdanerlt — 2 contributionsdermodmaster — 2 contributionsdhinkris — 2 contributionsdonaldwasserman — 2 contributionsdunnkers — 2 contributionskbarendrecht — 2 contributionskhannurien — 2 contributionskinga-marszalkowska — 2 contributionskonerzajakub — 2 contributionskrishna0125 — 2 contributionslouisbrulenaudet — 2 contributionsLucasLeRay — 2 contributionslwarsaame — 2 contributionsmarr75 — 2 contributionsmdsalnikov — 2 contributionsmikkeyboi — 2 contributionsnabeel-chhatri — 2 contributionsNikyParfenov — 2 contributionsoftenfrequent — 2 contributionsPradyMagal — 2 contributionsraphaeluzan — 2 contributionsrohinish404 — 2 contributionsRussell-Day — 2 contributionsS3lc0uth — 2 contributionssisp — 2 contributionssobs0 — 2 contributionstiffanychum — 2 contributionsyudhiesh — 2 contributionsyujiiroo — 2 contributions88roy88 — 1 contributiona-romero — 1 contributionAaryanverma — 1 contributionacompa — 1 contributionadityamehra — 1 contributionagent-kira — 1 contributionAjay6601 — 1 contributionAmaliMatharaarachchi — 1 contributionAMindToThink — 1 contributionamrakshay — 1 contributionAugmentMo — 1 contributionbmerkle — 1 contributionbowenliang123 — 1 contributioncancelself — 1 contributioncarvalho28 — 1 contributioncastelo-software — 1 contributionchaliy — 1 contributionchkimes — 1 contributioncmorris108 — 1 contributionconnorbrinton — 1 contributioncss911 — 1 contributionDanielYakubov — 1 contributiondebangshu919 — 1 contributionDeeds67 — 1 contributiondendarrion — 1 contributiondenis-snyk — 1 contributionderickson — 1 contributionDevilsAutumn — 1 contributiondhanesh24g — 1 contributiondmtri35 — 1 contributiondokato — 1 contributiondowithless — 1 contributiondufraux-adrien-m — 1 contributionDylanLi-Hang — 1 contributionebjaime — 1 contributioneduardoarndt — 1 contributioneLafo — 1 contributioneltociear — 1 contributionexhyy — 1 contributionfabiofumarola — 1 contributionfangshengren — 1 contributionfedesierr — 1 contributionFilippoPaganelli — 1 contributionfj11 — 1 contributionftnext — 1 contributiongavmor — 1 contributiongrant-sobkowski — 1 contributionhimanshutech4purpose — 1 contributionhimanushi — 1 contributionimanousar — 1 contributionj-mesnil — 1 contributionj1z0 — 1 contributionjaime-cespedes-sisniega — 1 contributionjakelucasnyc — 1 contributionjalling97 — 1 contributionjaywyawhare — 1 contributionJerry-Terrasse — 1 contributionJevDev2304 — 1 contributionjhs — 1 contributionji21 — 1 contributionjnchen — 1 contributionJohanCifuentes03 — 1 contributionJonasHildershavnUke — 1 contributionjrnt30 — 1 contributionjschomay — 1 contributionkarthick965938 — 1 contributionKelp710 — 1 contributionknulpi — 1 contributionKolodziejczykWaldemar — 1 contributionkoriyoshi2041 — 1 contributionkubre — 1 contributionkucharzyk-sebastian — 1 contributionLads-oxygen — 1 contributionlbux — 1 contributionlicux — 1 contributionlkacenja — 1 contributionlukmanarifs — 1 contributionMANISH007700 — 1 contributionmeroo36 — 1 contributionmeteatamel — 1 contributionmfaizanse — 1 contributionMizuki8783 — 1 contributionmoruga123 — 1 contributionmrazizi — 1 contributionnavkar98 — 1 contributionNeelayS — 1 contributionnicholasburka — 1 contributionnictuku — 1 contributionnimishbongale — 1 contributionNimJay — 1 contributionnishant-mahesh — 1 contributionniyasrad — 1 contributionnkhus — 1 contributionnoah-gil — 1 contributionnsking02 — 1 contributionottingbob — 1 contributionOwenKephart — 1 contributionp-constant — 1 contributionpavan555 — 1 contributionphilipchung — 1 contributionphilnash — 1 contributionPLNech — 1 contributionpomcho555 — 1 contributionpranay0703 — 1 contributionpritamsoni-hsr — 1 contributionPropetHI — 1 contributionqige96 — 1 contributionr-sniper — 1 contributionRajRavi05 — 1 contributionRasputin2 — 1 contributionrealei — 1 contributionreasonmethis — 1 contributionrepetitioestmaterstudiorum — 1 contributionRinZ27 — 1 contributionRishiSankineni — 1 contributionrohit-clearspot-ai — 1 contributionrouge8 — 1 contributionSe-Hun — 1 contributionseorc — 1 contributionshrimpnoodles — 1 contributionshun-liang — 1 contributionSighingSnow — 1 contributionsimon376 — 1 contributionsimoneb — 1 contributionsipa-echo-ngbm — 1 contributionskirdey-inflection — 1 contributionsnsk — 1 contributionStefanMojsilovic — 1 contributionSzymonCogiel — 1 contributiontanayag — 1 contributionTheNeuAra — 1 contributionthohag — 1 contributiontonton-golio — 1 contributiontyler-ball — 1 contributionudaykiran2427 — 1 contributionvection — 1 contributionVishnu-sai-teja — 1 contributionvmesel — 1 contributionwey-gu — 1 contributionwjfu99 — 1 contributionxiaopeiwu — 1 contributionzyuanlim — 1 contribution

Ah yes, FAQs.

This is the CTA :)

Start Evaluating