Available Benchmarks
DROP
DROP (Discrete Reasoning Over Paragraphs) is a benchmark designed to evaluate language models' advanced reasoning capabilities through complex question answering tasks. It encompasses over 9500 intricate challenges that demand numerical manipulations, multi-step reasoning, and the interpretation of text-based data. For more insights and access to the dataset, you can read the original DROP paper here.
Arguments
There are TWO optional arguments when using the DROP benchmark:
- [Optional]
tasks: a list of tasks (DROPTaskenums), which specifies the subject areas for model evaluation. By default, this is set to all tasks. The list ofDROPTaskenums can be found here. - [Optional]
n_shots: the number of examples for few-shot learning. This is set to 5 by default and cannot exceed 5.
Usage
The code below assesses a custom mistral_7b model (click here to learn how to use ANY custom LLM) on HISTORY_1002 and NFL_649 in DROP using 3-shot prompting.
from deepeval.benchmarks import DROP
from deepeval.benchmarks.tasks import DROPTask
# Define benchmark with specific tasks and shots
benchmark = DROP(
tasks=[DROPTask.HISTORY_1002, DROPTask.NFL_649],
n_shots=3
)
# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on exact matching, is calculated by determining the proportion of questions for which the model produces the precise correct answer (e.g. '3' or ‘John Doe’) in relation to the total number of questions.
As a result, utilizing more few-shot prompts (n_shots) can greatly improve the model's robustness in generating answers in the exact correct format and boost the overall score.
DROP Tasks
The DROPTask enum classifies the diverse range of categories covered in the DROP benchmark.
from deepeval.benchmarks.tasks import DROPTask
drop_tasks = [NFL_649]Below is the comprehensive list of available tasks:
NFL_649HISTORY_1418HISTORY_75HISTORY_2785NFL_227NFL_2684HISTORY_1720NFL_1333HISTORY_221HISTORY_2090HISTORY_241HISTORY_2951HISTORY_3897HISTORY_1782HISTORY_4078NFL_692NFL_104NFL_899HISTORY_2641HISTORY_3628HISTORY_488NFL_46HISTORY_752HISTORY_1262HISTORY_4118HISTORY_1425HISTORY_460NFL_1962HISTORY_1308NFL_969NFL_317HISTORY_370HISTORY_1837HISTORY_2626NFL_987NFL_87NFL_2996NFL_2082HISTORY_23HISTORY_787HISTORY_405HISTORY_1401HISTORY_835HISTORY_565HISTORY_1998HISTORY_2176HISTORY_1196HISTORY_1237NFL_244HISTORY_3109HISTORY_1414HISTORY_2771HISTORY_3806NFL_1233NFL_802HISTORY_2270NFL_578HISTORY_1313NFL_1216NFL_256HISTORY_3356HISTORY_1859HISTORY_3103HISTORY_2991HISTORY_2060HISTORY_1408HISTORY_3042NFL_1873NFL_1476NFL_524HISTORY_1316HISTORY_1456HISTORY_104HISTORY_1275HISTORY_1069NFL_3270NFL_1222HISTORY_2704HISTORY_733NFL_1981NFL_592HISTORY_920HISTORY_951NFL_1136HISTORY_2642HISTORY_1065HISTORY_2976NFL_669HISTORY_2846NFL_1996HISTORY_2848NFL_3285HISTORY_2789HISTORY_3722HISTORY_514HISTORY_869HISTORY_2857HISTORY_3237NFL_563HISTORY_990HISTORY_2961NFL_3387HISTORY_124HISTORY_2898HISTORY_2925HISTORY_2788HISTORY_632HISTORY_2619HISTORY_3278NFL_749HISTORY_3726NFL_1096NFL_1207HISTORY_3079HISTORY_2939HISTORY_3581NFL_2777HISTORY_3873HISTORY_1731HISTORY_426NFL_1478HISTORY_3106NFL_1498NFL_3133HISTORY_3345NFL_503HISTORY_801NFL_2931NFL_2482HISTORY_1945NFL_2262HISTORY_3735HISTORY_1151NFL_2415HISTORY_607HISTORY_724HISTORY_1284HISTORY_494NFL_3571NFL_1307HISTORY_2847HISTORY_2650NFL_1586NFL_2478HISTORY_1276NFL_540NFL_894NFL_1492HISTORY_3265HISTORY_686HISTORY_2546NFL_2396HISTORY_2001HISTORY_1793HISTORY_2014HISTORY_2732HISTORY_2927NFL_1195HISTORY_1650NFL_2077HISTORY_3036HISTORY_495HISTORY_3048HISTORY_912HISTORY_936NFL_1329HISTORY_1928HISTORY_3303HISTORY_2199HISTORY_1169HISTORY_115HISTORY_2575HISTORY_1340NFL_988HISTORY_423HISTORY_1959NFL_29HISTORY_2867NFL_2191HISTORY_3754NFL_1021NFL_2269HISTORY_4060HISTORY_1773HISTORY_2757HISTORY_468HISTORY_10HISTORY_2151HISTORY_725NFL_858NFL_122HISTORY_591HISTORY_2948HISTORY_2829HISTORY_4034HISTORY_3717HISTORY_187HISTORY_1995NFL_1566HISTORY_685HISTORY_296HISTORY_1876HISTORY_2733HISTORY_325HISTORY_1898HISTORY_1948NFL_1838HISTORY_3993HISTORY_3366HISTORY_79NFL_2584HISTORY_3241HISTORY_1879HISTORY_2004HISTORY_4050NFL_2668HISTORY_3683HISTORY_836HISTORY_783HISTORY_2953HISTORY_1723NFL_378HISTORY_4137HISTORY_200HISTORY_502HISTORY_175HISTORY_3341HISTORY_2196HISTORY_9NFL_2385NFL_1879HISTORY_1298NFL_2272HISTORY_2170HISTORY_4080HISTORY_3669HISTORY_3647HISTORY_586NFL_1454HISTORY_2760HISTORY_1498HISTORY_1415HISTORY_2361NFL_915HISTORY_986HISTORY_1744HISTORY_1802HISTORY_3075HISTORY_2412NFL_832HISTORY_3435HISTORY_1306HISTORY_3089HISTORY_1002HISTORY_3949HISTORY_1445HISTORY_254HISTORY_991HISTORY_2530HISTORY_447HISTORY_2661HISTORY_1746HISTORY_347NFL_3009HISTORY_1814NFL_3126HISTORY_972NFL_2528HISTORY_2417NFL_1184HISTORY_59HISTORY_1811HISTORY_3115HISTORY_71HISTORY_1935HISTORY_2944HISTORY_1019HISTORY_887HISTORY_533NFL_3195HISTORY_3615HISTORY_4007HISTORY_2950NFL_1672HISTORY_2897HISTORY_1887HISTORY_2836NFL_3356HISTORY_1828HISTORY_3714NFL_2054HISTORY_2709NFL_1883NFL_2042HISTORY_2162NFL_2197NFL_2369HISTORY_2765HISTORY_2021NFL_1152HISTORY_2957HISTORY_1863HISTORY_2064HISTORY_4045HISTORY_3058NFL_153HISTORY_1074HISTORY_159HISTORY_455HISTORY_761HISTORY_1552NFL_1769NFL_880NFL_2234NFL_2995NFL_2823HISTORY_2179HISTORY_1891HISTORY_2474HISTORY_3062NFL_490HISTORY_1416HISTORY_415HISTORY_2609NFL_1618HISTORY_3749HISTORY_68HISTORY_4011NFL_2067NFL_610NFL_2568NFL_1689HISTORY_2044HISTORY_1844HISTORY_3992NFL_716NFL_825HISTORY_806NFL_194HISTORY_2970HISTORY_2878NFL_1652HISTORY_3804HISTORY_90NFL_16HISTORY_515HISTORY_1954HISTORY_2011HISTORY_2832HISTORY_228NFL_2907HISTORY_2752HISTORY_1352HISTORY_3244HISTORY_2941HISTORY_1227HISTORY_130HISTORY_3587HISTORY_69HISTORY_2676NFL_1768NFL_995HISTORY_809HISTORY_941HISTORY_3264NFL_1264HISTORY_1012HISTORY_1450HISTORY_1048NFL_719HISTORY_2762HISTORY_2086HISTORY_1259NFL_1240HISTORY_2234HISTORY_2102HISTORY_688NFL_2114HISTORY_1459HISTORY_1043HISTORY_3609NFL_1223HISTORY_417HISTORY_1884HISTORY_2390NFL_2671HISTORY_2298HISTORY_659HISTORY_459HISTORY_1542NFL_1914HISTORY_1258HISTORY_2164HISTORY_2777NFL_1304HISTORY_4049HISTORY_1423NFL_2994HISTORY_2814HISTORY_2187HISTORY_3280HISTORY_794NFL_3342HISTORY_2153HISTORY_1708NFL_1540HISTORY_92HISTORY_1907NFL_290NFL_1167HISTORY_2885HISTORY_2258HISTORY_1940HISTORY_2380NFL_1245HISTORY_3552HISTORY_534NFL_1193NFL_264NFL_275HISTORY_1042NFL_1829NFL_2571NFL_296NFL_199HISTORY_2434NFL_1486HISTORY_107HISTORY_371NFL_1361HISTORY_1212NFL_2036NFL_913HISTORY_2886HISTORY_2737HISTORY_487NFL_1516NFL_2894HISTORY_3692NFL_496HISTORY_2707HISTORY_655NFL_286HISTORY_13HISTORY_556NFL_962HISTORY_1517HISTORY_1130NFL_624NFL_2125NFL_1670HISTORY_512NFL_1515HISTORY_893HISTORY_1233HISTORY_3116HISTORY_544HISTORY_3807HISTORY_2088NFL_2601HISTORY_1952HISTORY_131HISTORY_3662HISTORY_883HISTORY_2949HISTORY_1965NFL_778HISTORY_2047HISTORY_4009HISTORY_520HISTORY_1748HISTORY_154NFL_493NFL_187HISTORY_1578NFL_1344NFL_3489NFL_246NFL_336NFL_3396NFL_816NFL_1390HISTORY_3363HISTORY_4002HISTORY_4141NFL_1378HISTORY_476NFL_477NFL_1471NFL_3420HISTORY_227HISTORY_3859NFL_715HISTORY_283HISTORY_1943HISTORY_1665HISTORY_1860NFL_2387HISTORY_3253HISTORY_2766HISTORY_671HISTORY_720HISTORY_3141HISTORY_1373HISTORY_2453HISTORY_3608HISTORY_343NFL_2918HISTORY_3866HISTORY_2818NFL_2330NFL_2636NFL_1553HISTORY_1082HISTORY_3900NFL_2202HISTORY_3404HISTORY_103NFL_2409NFL_1412HISTORY_2188NFL_3386NFL_1503NFL_1288NFL_2151NFL_1743HISTORY_2815HISTORY_2671HISTORY_1892NFL_613HISTORY_1356HISTORY_2363HISTORY_424HISTORY_3438HISTORY_148NFL_3290NFL_663HISTORY_732HISTORY_3092HISTORY_408NFL_3460HISTORY_2809HISTORY_530HISTORY_3588HISTORY_1853HISTORY_513HISTORY_918HISTORY_908HISTORY_2869HISTORY_1125HISTORY_796HISTORY_1601HISTORY_1250HISTORY_1092HISTORY_351HISTORY_2142NFL_2255HISTORY_3533HISTORY_3400HISTORY_2456HISTORY_3164HISTORY_2339NFL_2297HISTORY_3105NFL_1596NFL_2893HISTORY_539NFL_1332HISTORY_208NFL_350NFL_2645HISTORY_2921HISTORY_1167HISTORY_2892HISTORY_791NFL_3222NFL_1789NFL_180NFL_3594HISTORY_3143NFL_824NFL_2034
FAQs
What does the DROP benchmark measure?
DROP (Discrete Reasoning Over Paragraphs) evaluates an LLM's advanced reasoning through complex question answering. It encompasses over 9,500 challenges that demand numerical reasoning (addition, subtraction, counting), multi-step reasoning, and the interpretation of text-based data drawn from paragraphs about NFL and history.
Which tasks can I run with DROP?
You can pass a list of
DROPTask enums to the tasks argument, for example HISTORY_1002 or NFL_649. By default, all available tasks are evaluated.How is DROP scored?
The
overall_score ranges from 0 to 1 and is based on exact matching: it is the proportion of questions for which the model produces the precise correct answer (e.g. '3' or 'John Doe'). See the benchmarks introduction for how scoring works across benchmarks.What is the default n_shots for DROP?
The
n_shots argument is set to 5 by default and cannot exceed 5. Using more few-shot prompts can improve the model's robustness in generating answers in the exact correct format.Does DROP support chain-of-thought prompting?
No. Unlike
BigBenchHard, there is no CoT prompting for the DROP benchmark. Its only optional arguments are tasks and n_shots.How do I run DROP on a custom LLM?
Wrap your model in
DeepEvalBaseLLM, create a benchmark with DROP(tasks=[...], n_shots=3), and call benchmark.evaluate(model=mistral_7b) before reading benchmark.overall_score. See the custom LLM guide for details.