SQuAD
SQuAD (Stanford Question Answering Dataset) is a QA benchmark designed to test a language model's reading comprehension capabilities. It consists of 100K question-answer pairs (including 10K in the validation set), where each answer is a segment of text taken directly from the accompanying reading passage. To learn more about the dataset and its construction, you can read the original SQuAD paper here.
Arguments
There are THREE optional arguments when using the SQuAD benchmark:
- [Optional]
tasks: a list of tasks (SQuADTaskenums), which specifies the subject areas for model evaluation. By default, this is set to all tasks. The list ofSQuADTaskenums can be found here. - [Optional]
n_shots: the number of examples for few-shot learning. This is set to 5 by default and cannot exceed 5. - [Optional]
evaluation_model: a string specifying which of OpenAI's GPT models to use for scoring, OR any custom LLM model of typeDeepEvalBaseLLM. Defaulted togpt-5.4.
Usage
The code below assesses a custom mistral_7b model (click here to learn how to use ANY custom LLM) on passages about pharmacy and Normans in SQuAD using 3-shot prompting.
from deepeval.benchmarks import SQuAD
from deepeval.benchmarks.tasks import SQuADTask
# Define benchmark with specific tasks and shots
benchmark = SQuAD(
tasks=[SQuADTask.PHARMACY, SQuADTask.NORMANS],
n_shots=3
)
# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on LLM-as-a-judge, is calculated by evaluating whether the predicted answer aligns with the expected output based on the passage context.
For example, if the question asks, "How many atoms are present?" and the model predicts "two atoms," the LLM-as-a-judge determines whether this aligns with the expected answer of "2" by assessing semantic equivalence rather than exact text matching.
SQuAD Tasks
The SQuADTask enum classifies the diverse range of categories covered in the SQuAD benchmark.
from deepeval.benchmarks.tasks import SQuADTask
math_qa_tasks = [SQuADTask.PHARMACY]Below is the comprehensive list of available tasks:
PHARMACYNORMANSHUGUENOTDOCTOR_WHOOIL_CRISIS_1973COMPUTATIONAL_COMPLEXITY_THEORYWARSAWAMERICAN_BROADCASTING_COMPANYCHLOROPLASTAPOLLO_PROGRAMTEACHERMARTIN_LUTHERECONOMIC_INEQUALITYYUAN_DYNASTYSCOTTISH_PARLIAMENTISLAMISMUNITED_METHODIST_CHURCHIMMUNE_SYSTEMNEWCASTLE_UPON_TYNECTENOPHORAFRESNO_CALIFORNIASTEAM_ENGINEPACKET_SWITCHINGFORCEJACKSONVILLE_FLORIDAEUROPEAN_UNION_LAWSUPER_BOWL_50VICTORIA_AND_ALBERT_MUSEUMBLACK_DEATHCONSTRUCTIONSKY_UKUNIVERSITY_OF_CHICAGOVICTORIA_AUSTRALIAFRENCH_AND_INDIAN_WARIMPERIALISMPRIVATE_SCHOOLGEOLOGYHARVARD_UNIVERSITYRHINEPRIME_NUMBERINTERGOVERNMENTAL_PANEL_ON_CLIMATE_CHANGEAMAZON_RAINFORESTKENYASOUTHERN_CALIFORNIANIKOLA_TESLACIVIL_DISOBEDIENCEGENGHIS_KHANOXYGEN
FAQs
What does the SQuAD benchmark measure?
How is SQuAD scored in deepeval?
Why does SQuAD require an evaluation_model?
evaluation_model. It accepts a string for one of OpenAI's GPT models, or any custom LLM of type DeepEvalBaseLLM, and the judge decides whether a prediction and the expected output align.What is the default n_shots for SQuAD?
5 and cannot exceed that. Use the n_shots argument to lower it for few-shot prompting.Which tasks can I evaluate with SQuAD?
SQuADTask enums to the tasks argument to target specific subject areas such as PHARMACY or NORMANS. By default, all tasks are evaluated.How do I run SQuAD on a custom LLM?
evaluate() with your model (for example a custom DeepEvalBaseLLM) and read overall_score, which ranges from 0 to 1. See benchmarking your LLM for details.