Available Benchmarks
HellaSwag
HellaSwag is a benchmark designed to evaluate language models' commonsense reasoning through sentence completion tasks. It provides 10,000 challenges spanning various subject areas. For more details, you can visit the Hellaswag GitHub page.
Arguments
There are TWO optional arguments when using the HellaSwag benchmark:
- [Optional]
tasks: a list of tasks (HellaSwagTaskenums), which specifies the subject areas for sentence completion evaluation. By default, this is set to all tasks. The list ofHellaSwagTaskenums can be found here. - [Optional]
n_shots: the number of "shots" to use for few-shot learning. This is set to 10 by default and cannot exceed 15.
Usage
The code below evaluates a custom mistral_7b model (click here to learn how to use ANY custom LLM) and its ability to complete sentences related to 'Trimming Branches or Hedges' and 'Baton Twirling' subjects using 5-shot learning.
from deepeval.benchmarks import HellaSwag
from deepeval.benchmarks.tasks import HellaSwagTask
# Define benchmark with specific tasks and shots
benchmark = HellaSwag(
tasks=[HellaSwagTask.TRIMMING_BRANCHES_OR_HEDGES, HellaSwagTask.BATON_TWIRLING],
n_shots=5
)
# Replace 'mistral_7b' with your own custom model
benchmark.evaluate(model=mistral_7b)
print(benchmark.overall_score)The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on exact matching, is calculated by determining the proportion of multiple-choice sentence-completion questions for which the model produces the precise correct letter answer (e.g. 'A') in relation to the total number of questions.
As a result, utilizing more few-shot prompts (n_shots) can greatly improve the model's robustness in generating answers in the exact correct format and boost the overall score.
HellaSwag Tasks
The HellaSwagTask enum classifies the diverse range of categories covered in the HellaSwag benchmark.
from deepeval.benchmarks.tasks import HellaSwagTask
hella_tasks = [HellaSwagTask.APPLYING_SUNSCREEN]Below is the comprehensive list of available tasks:
APPLYING_SUNSCREENTRIMMING_BRANCHES_OR_HEDGESDISC_DOGWAKEBOARDINGSKATEBOARDINGWATERSKIINGWASHING_HANDSSAILINGPLAYING_CONGASBALLETROOF_SHINGLE_REMOVALHAND_CAR_WASHKITE_FLYINGPLAYING_POOLPLAYING_LACROSSELAYUP_DRILL_IN_BASKETBALLHOME_AND_GARDENPLAYING_BEACH_VOLLEYBALLCALF_ROPINGSCUBA_DIVINGMIXING_DRINKSPUTTING_ON_SHOESMAKING_A_LEMONADEUNCATEGORIZEDZUMBAPLAYING_BADMINTONPLAYING_BAGPIPESFOOD_AND_ENTERTAININGPERSONAL_CARE_AND_STYLECRICKETSHOVELING_SNOWPING_PONGHOLIDAYS_AND_TRADITIONSICE_FISHINGBEACH_SOCCERTABLE_SOCCERSWIMMINGBATON_TWIRLINGJAVELIN_THROWSHOT_PUTDOING_CRUNCHESPOLISHING_SHOESTRAVELUSING_UNEVEN_BARSPLAYING_HARMONICARELATIONSHIPSHIGH_JUMPMAKING_A_SANDWICHPOWERBOCKINGREMOVING_ICE_FROM_CARSHAVINGSHARPENING_KNIVESWELDINGUSING_PARALLEL_BARSHOME_CATEGORIESROCK_CLIMBINGSNOW_TUBINGWASHING_FACEASSEMBLING_BICYCLETENNIS_SERVE_WITH_BALL_BOUNCINGSHUFFLEBOARDDODGEBALLCAPOEIRAPAINTBALLDOING_A_POWERBOMBDOING_MOTOCROSSPLAYING_ICE_HOCKEYPHILOSOPHY_AND_RELIGIONARCHERYCARS_AND_OTHER_VEHICLESRUNNING_A_MARATHONTHROWING_DARTSPAINTING_FURNITUREHAVING_AN_ICE_CREAMSLACKLININGCAMEL_RIDEARM_WRESTLINGHULA_HOOPSURFINGPLAYING_PIANOGARGLING_MOUTHWASHPLAYING_ACCORDIONHORSEBACK_RIDINGPUTTING_IN_CONTACT_LENSESPLAYING_SAXOPHONEFUTSALLONG_JUMPLONGBOARDINGPOLE_VAULTBUILDING_SANDCASTLESPLATFORM_DIVINGPAINTINGSPINNINGCARVING_JACK_O_LANTERNSBRAIDING_HAIRYOUTHPLAYING_VIOLINCANOEINGCHEERLEADINGPETS_AND_ANIMALSKAYAKINGCLEANING_SHOESKNITTINGBAKING_COOKIESDOING_FENCINGPLAYING_GUITARRAUSING_THE_ROWING_MACHINEGETTING_A_HAIRCUTMOOPING_FLOORRIVER_TUBINGCLEANING_SINKGROOMING_DOGDISCUS_THROWCLEANING_WINDOWSFINANCE_AND_BUSINESSHANGING_WALLPAPERROPE_SKIPPINGWINDSURFINGKNEELINGGETTING_A_PIERCINGROCK_PAPER_SCISSORSSPORTS_AND_FITNESSBREAKDANCINGWALKING_THE_DOGPLAYING_DRUMSPLAYING_WATER_POLOBMXSMOKING_A_CIGARETTEBLOWING_LEAVESBULLFIGHTINGDRINKING_COFFEEBATHING_DOGTANGOWRAPPING_PRESENTSPLASTERINGPLAYING_BLACKJACKFUN_SLIDING_DOWNWORK_WORLDTRIPLE_JUMPTUMBLINGSKIINGDOING_KICKBOXINGBLOW_DRYING_HAIRDRUM_CORPSSMOKING_HOOKAHMOWING_THE_LAWNVOLLEYBALLLAYING_TILESTARTING_A_CAMPFIRESUMOHURLINGPLAYING_KICKBALLMAKING_A_CAKEFIXING_THE_ROOFPLAYING_POLOREMOVING_CURLERSELLIPTICAL_TRAINERHEALTHSPREAD_MULCHCHOPPING_WOODBRUSHING_TEETHUSING_THE_POMMEL_HORSESNATCHCLIPPING_CAT_CLAWSPUTTING_ON_MAKEUPHAND_WASHING_CLOTHESHITTING_A_PINATATAI_CHIGETTING_A_TATTOODRINKING_BEERSHAVING_LEGSDOING_KARATEPLAYING_RUBIK_CUBEFAMILY_LIFEROLLERBLADINGEDUCATION_AND_COMMUNICATIONSFIXING_BICYCLEBEER_PONGIRONING_CLOTHESCUTTING_THE_GRASSRAKING_LEAVESPLAYING_SQUASHHOPSCOTCHINSTALLING_CARPETPOLISHING_FURNITUREDECORATING_THE_CHRISTMAS_TREEPREPARING_SALADPREPARING_PASTAVACUUMING_FLOORCLEAN_AND_JERKCOMPUTERS_AND_ELECTRONICSCROQUET
FAQs
What does the HellaSwag benchmark measure?
HellaSwag evaluates an LLM's commonsense reasoning through sentence completion tasks. It provides 10,000 challenges spanning various subject areas, making it useful for pinpointing where models struggle with nuanced or complex contexts.Which tasks can I run with HellaSwag?
You can pass a list of
HellaSwagTask enums to the tasks argument, for example TRIMMING_BRANCHES_OR_HEDGES or BATON_TWIRLING. By default, all available tasks are evaluated.How is HellaSwag scored?
The
overall_score ranges from 0 to 1 and is based on exact matching: it is the proportion of multiple-choice sentence-completion questions for which the model produces the precise correct letter answer (e.g. 'A'). See the benchmarks introduction for how scoring works across benchmarks.What is the default n_shots for HellaSwag?
The
n_shots argument is set to 10 by default and cannot exceed 15. Using more few-shot prompts can improve the model's robustness in generating answers in the exact correct format.Does HellaSwag support chain-of-thought prompting?
No. Unlike
BigBenchHard, there is no CoT prompting for the HellaSwag benchmark. Its only optional arguments are tasks and n_shots.How do I run HellaSwag on a custom LLM?
Wrap your model in
DeepEvalBaseLLM, create a benchmark with HellaSwag(tasks=[...], n_shots=5), and call benchmark.evaluate(model=mistral_7b) before reading benchmark.overall_score. See the benchmarking guide for using any custom LLM.