MMLU
MMLU (Massive Multitask Language Understanding) is a benchmark for evaluating LLMs through multiple-choice questions. These questions cover 57 subjects such as math, history, law, and ethics. For more information, visit the MMLU GitHub page.
Arguments
There are TWO optional arguments when using the MMLU benchmark:
- [Optional]
tasks: a list of tasks (MMLUTaskenums), specifying which of the 57 subject areas to evaluate in the language model. By default, this is set to all tasks. Detailed descriptions of theMMLUTaskenum can be found here. - [Optional]
n_shots: the number of "shots" to use for few-shot learning. This is set to 5 by default and cannot exceed this number.
Usage
The code below evaluates a custom mistral_7b model (click here to learn how to use ANY custom LLM) and assesses its performance on High School Computer Science and Astronomy using 3-shot learning.
from deepeval.benchmarks import MMLU
from deepeval.benchmarks.mmlu.task import MMLUTask
# Define benchmark with specific tasks and shots
benchmark = MMLU(
tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE, MMLUTask.ASTRONOMY],
n_shots=3
)
# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on exact matching, is calculated by determining the proportion of multiple-choice questions for which the model produces the precise correct letter answer (e.g. 'A') in relation to the total number of questions.
As a result, utilizing more few-shot prompts (n_shots) can greatly improve the model's robustness in generating answers in the exact correct format and boost the overall score.
MMLU Tasks
The MMLUTask enum classifies the diverse range of subject areas covered in the MMLU benchmark.
from deepeval.benchmarks.tasks import MMLUTask
mm_tasks = [MMLUTask.HIGH_SCHOOL_EUROPEAN_HISTORY]Below is the comprehensive list of all available tasks:
HIGH_SCHOOL_EUROPEAN_HISTORYBUSINESS_ETHICSCLINICAL_KNOWLEDGEMEDICAL_GENETICSHIGH_SCHOOL_US_HISTORYHIGH_SCHOOL_PHYSICSHIGH_SCHOOL_WORLD_HISTORYVIROLOGYHIGH_SCHOOL_MICROECONOMICSECONOMETRICSCOLLEGE_COMPUTER_SCIENCEHIGH_SCHOOL_BIOLOGYABSTRACT_ALGEBRAPROFESSIONAL_ACCOUNTINGPHILOSOPHYPROFESSIONAL_MEDICINENUTRITIONGLOBAL_FACTSMACHINE_LEARNINGSECURITY_STUDIESPUBLIC_RELATIONSPROFESSIONAL_PSYCHOLOGYPREHISTORYANATOMYHUMAN_SEXUALITYCOLLEGE_MEDICINEHIGH_SCHOOL_GOVERNMENT_AND_POLITICSCOLLEGE_CHEMISTRYLOGICAL_FALLACIESHIGH_SCHOOL_GEOGRAPHYELEMENTARY_MATHEMATICSHUMAN_AGINGCOLLEGE_MATHEMATICSHIGH_SCHOOL_PSYCHOLOGYFORMAL_LOGICHIGH_SCHOOL_STATISTICSINTERNATIONAL_LAWHIGH_SCHOOL_MATHEMATICSHIGH_SCHOOL_COMPUTER_SCIENCECONCEPTUAL_PHYSICSMISCELLANEOUSHIGH_SCHOOL_CHEMISTRYMARKETINGPROFESSIONAL_LAWMANAGEMENTCOLLEGE_PHYSICSJURISPRUDENCEWORLD_RELIGIONSSOCIOLOGYUS_FOREIGN_POLICYHIGH_SCHOOL_MACROECONOMICSCOMPUTER_SECURITYMORAL_SCENARIOSMORAL_DISPUTESELECTRICAL_ENGINEERINGASTRONOMYCOLLEGE_BIOLOGY
FAQs
What does the MMLU benchmark measure?
MMLU (Massive Multitask Language Understanding) evaluates an LLM through multiple-choice questions spanning 57 subjects such as math, history, law, and ethics. It is good at detecting areas where a model may lack understanding in a particular topic.Which tasks can I run with MMLU?
MMLUTask enums via the tasks argument, for example HIGH_SCHOOL_COMPUTER_SCIENCE or ASTRONOMY. By default, deepeval evaluates your LLM on all 57 subject areas.How is MMLU scored?
overall_score ranges from 0 to 1 and is based on exact matching: it is the proportion of multiple-choice questions for which the model produces the precise correct letter answer (e.g. 'A'). See the benchmarks introduction for how scoring works across benchmarks.What is the default n_shots for MMLU?
n_shots argument defaults to 5 and cannot exceed that number. Using more few-shot prompts can improve the model's robustness in generating answers in the exact correct format and boost the overall score.Does MMLU support chain-of-thought prompting?
MMLU benchmark exposes only two optional arguments, tasks and n_shots, and does not include an enable_cot option for chain-of-thought prompting.How do I run MMLU on a custom LLM?
DeepEvalBaseLLM, create a benchmark with MMLU(tasks=[...], n_shots=3), and call benchmark.evaluate(model=mistral_7b). You can then read benchmark.overall_score. See the benchmarking guide for using any custom LLM.