LogiQA
LogiQA is a comprehensive dataset designed to assess an LLM's logical reasoning capabilities, encompassing various types of deductive reasoning, including categorical and disjunctive reasoning. It features 8,678 multiple-choice questions, each paired with a reading passage. To learn more about the dataset and its construction, you can read the original paper here.
Arguments
There are TWO optional arguments when using the LogiQA benchmark:
- [Optional]
tasks: a list of tasks (LogiQATaskenums), which specifies the subject areas for model evaluation. By default, this is set to all tasks. The list ofLogiQATaskenums can be found here. - [Optional]
n_shots: the number of examples for few-shot learning. This is set to 5 by default and cannot exceed 5.
Usage
The code below assesses a custom mistral_7b model (click here to learn how to use ANY custom LLM) on categorical reasoning and sufficient conditional reasoning using 3-shot prompting.
from deepeval.benchmarks import LogiQA
from deepeval.benchmarks.tasks import LogiQATask
# Define benchmark with specific tasks and shots
benchmark = LogiQA(
tasks=[LogiQATask.CATEGORICAL_REASONING, LogiQATask.SUFFICIENT_CONDITIONAL_REASONING],
n_shots=3
)
# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on exact matching, is calculated by determining the proportion of questions for which the model produces the precise correct multiple choice answer (e.g. 'A' or ‘C’) in relation to the total number of questions.
LogiQA Tasks
The LogiQATask enum classifies the diverse range of reasoning categories covered in the LogiQA benchmark.
from deepeval.benchmarks.tasks import LogiQATask
math_qa_tasks = [LogiQATask.CATEGORICAL_REASONING]Below is the comprehensive list of available tasks:
CATEGORICAL_REASONINGSUFFICIENT_CONDITIONAL_REASONINGNECESSARY_CONDITIONAL_REASONINGDISJUNCTIVE_REASONINGCONJUNCTIVE_REASONING
FAQs
What does the LogiQA benchmark measure?
How is LogiQA scored?
overall_score is the proportion of questions for which the model produces the precise correct multiple-choice answer (e.g. 'A' or 'C') out of the total. It ranges from 0 to 1, where 1 signifies perfect performance.What is the default n_shots for LogiQA?
5 and cannot exceed that. Use the n_shots argument to lower it for few-shot prompting.Which reasoning tasks can I evaluate with LogiQA?
LogiQATask enums to the tasks argument. Available tasks are CATEGORICAL_REASONING, SUFFICIENT_CONDITIONAL_REASONING, NECESSARY_CONDITIONAL_REASONING, DISJUNCTIVE_REASONING, and CONJUNCTIVE_REASONING. By default, all tasks are evaluated.Can using more few-shot prompts improve the LogiQA score?
n_shots can greatly improve the model's robustness in generating answers in the exact correct format and boost the overall_score.How do I run LogiQA on a custom LLM?
evaluate() with your model and read overall_score. See benchmarking your LLM to learn how to use any custom LLM.