HumanEval
The HumanEval benchmark is a dataset designed to evaluate an LLM’s code generation capabilities. The benchmark consists of 164 hand-crafted programming challenges comparable to simple software interview questions. For more information, visit the HumanEval GitHub page.
Arguments
There are TWO optional arguments when using the HumanEval benchmark:
- [Optional]
tasks: a list of tasks (HumanEvalTaskenums), specifying which of the 164 programming tasks to evaluate in the language model. By default, this is set to all tasks. Detailed descriptions of theHumanEvalTaskenum can be found here. - [Optional]
n: the number of code generation samples for each task for model evaluation using the pass@k metric. This is set to 200 by default. A more detailed description of thepass@kmetric andnparameter can be found here.
Usage
The code below evaluates a custom GPT-4 model (click here to learn how to use ANY custom LLM) and assesses its performance on HAS_CLOSE_ELEMENTS and SORT_NUMBERS tasks using 100 code generation samples.
from deepeval.benchmarks import HumanEval
from deepeval.benchmarks.tasks import HumanEvalTask
# Define benchmark with specific tasks and number of code generations
benchmark = HumanEval(
tasks=[HumanEvalTask.HAS_CLOSE_ELEMENTS, HumanEvalTask.SORT_NUMBERS],
n=100
)
# Replace 'gpt_4' with your own custom model
benchmark.evaluate(model=gpt_4, k=10)
print(benchmark.overall_score)You must define a generate_samples method in your custom model to perform HumanEval evaluation. In addition, when calling evaluate, you must supply k, the number of top samples chosen for the pass@k metric.
# Define a custom GPT-4 model class
class GPT4Model(DeepEvalBaseLLM):
...
def generate_samples(
self, prompt: str, n: int, temperature: float
) -> Tuple[AIMessage, float]:
chat_model = self.load_model()
og_parameters = {"n": chat_model.n, "temp": chat_model.temperature}
chat_model.n = n
chat_model.temperature = temperature
generations = chat_model._generate([HumanMessage(prompt)]).generations
completions = [r.text for r in generations]
return completions
...
gpt_4 = GPT4Model()The overall_score for this benchmark ranges from 0 to 1, where 1 signifies perfect performance and 0 indicates no correct answers. The model's score, based on the pass@k metric, is calculated by determining the proportion of code generations for which the model passes all the test cases (7.7 test cases average per problem) for at least k samples in relation to the total number of questions.
Pass@k Metric
The pass@k metric evaluates the functional correctness of generated code samples by focusing on whether at least one of the top k samples passes predefined unit tests. It calculates this probability by determining the complement of the probability that all k chosen samples are incorrect, using the formula:
where C represents combinations, n is the total number of samples, c is the number of correct samples, and k is the number of top samples chosen.
Using n helps ensure that the evaluation metric considers the full range of generated outputs, thereby reducing the risk of bias that can arise from only considering a small, possibly non-representative set of samples.
HumanEval Tasks
The HumanEvalTask enum classifies the diverse range of subject areas covered in the HumanEval benchmark.
from deepeval.benchmarks.tasks import HumanEvalTask
human_eval_tasks = [HumanEvalTask.HAS_CLOSE_ELEMENTS]Below is the comprehensive list of all available tasks:
HAS_CLOSE_ELEMENTSSEPARATE_PAREN_GROUPSTRUNCATE_NUMBERBELOW_ZEROMEAN_ABSOLUTE_DEVIATIONINTERSPERSEPARSE_NESTED_PARENSFILTER_BY_SUBSTRINGSUM_PRODUCTROLLING_MAXMAKE_PALINDROMESTRING_XORLONGESTGREATEST_COMMON_DIVISORALL_PREFIXESSTRING_SEQUENCECOUNT_DISTINCT_CHARACTERSPARSE_MUSICHOW_MANY_TIMESSORT_NUMBERSFIND_CLOSEST_ELEMENTSRESCALE_TO_UNITFILTER_INTEGERSSTRLENLARGEST_DIVISORFACTORIZEREMOVE_DUPLICATESFLIP_CASECONCATENATEFILTER_BY_PREFIXGET_POSITIVEIS_PRIMEFIND_ZEROSORT_THIRDUNIQUEMAX_ELEMENTFIZZ_BUZZSORT_EVENDECODE_CYCLICPRIME_FIBTRIPLES_SUM_TO_ZEROCAR_RACE_COLLISIONINCR_LISTPAIRS_SUM_TO_ZEROCHANGE_BASETRIANGLE_AREAFIB4MEDIANIS_PALINDROMEMODPDECODE_SHIFTREMOVE_VOWELSBELOW_THRESHOLDADDSAME_CHARSFIBCORRECT_BRACKETINGMONOTONICCOMMONLARGEST_PRIME_FACTORSUM_TO_NDERIVATIVEFIBFIBVOWELS_COUNTCIRCULAR_SHIFTDIGITSUMFRUIT_DISTRIBUTIONPLUCKSEARCHSTRANGE_SORT_LISTWILL_IT_FLYSMALLEST_CHANGETOTAL_MATCHIS_MULTIPLY_PRIMEIS_SIMPLE_POWERIS_CUBEHEX_KEYDECIMAL_TO_BINARYIS_HAPPYNUMERICAL_LETTER_GRADEPRIME_LENGTHSTARTS_ONE_ENDSSOLVEANTI_SHUFFLEGET_ROWSORT_ARRAYENCRYPTNEXT_SMALLESTIS_BOREDANY_INTENCODESKJKASDKDCHECK_DICT_CASECOUNT_UP_TOMULTIPLYCOUNT_UPPERCLOSEST_INTEGERMAKE_A_PILEWORDS_STRINGCHOOSE_NUMROUNDED_AVGUNIQUE_DIGITSBY_LENGTHEVEN_ODD_PALINDROMECOUNT_NUMSMOVE_ONE_BALLEXCHANGEHISTOGRAMREVERSE_DELETEODD_COUNTMINSUBARRAYSUMMAX_FILLSELECT_WORDSGET_CLOSEST_VOWELMATCH_PARENSMAXIMUMSOLUTIONADD_ELEMENTSGET_ODD_COLLATZVALID_DATESPLIT_WORDSIS_SORTEDINTERSECTIONPROD_SIGNSMINPATHTRIDIGITSIS_NESTEDSUM_SQUARESCHECK_IF_LAST_CHAR_IS_A_LETTERCAN_ARRANGELARGEST_SMALLEST_INTEGERSCOMPARE_ONEIS_EQUAL_TO_SUM_EVENSPECIAL_FACTORIALFIX_SPACESFILE_NAME_CHECKWORDS_IN_SENTENCESIMPLIFYORDER_BY_POINTSSPECIALFILTERGET_MAX_TRIPLESBFSORTED_LIST_SUMX_OR_YDOUBLE_THE_DIFFERENCECOMPARESTRONGEST_EXTENSIONCYCPATTERN_CHECKEVEN_ODD_COUNTINT_TO_MINI_ROMANRIGHT_ANGLE_TRIANGLEFIND_MAXEATDO_ALGEBRASTRING_TO_MD5GENERATE_INTEGERS
FAQs
What does the HumanEval benchmark measure?
HumanEval evaluates an LLM's code generation capabilities using 164 hand-crafted programming challenges comparable to simple software interview questions. It assesses the functional correctness of generated code rather than textual similarity to a reference solution.Which tasks can I run with HumanEval?
HumanEvalTask enums to the tasks argument, for example HAS_CLOSE_ELEMENTS or SORT_NUMBERS. By default, all 164 programming tasks are evaluated.How is HumanEval scored?
overall_score ranges from 0 to 1 and is based on the pass@k metric: the proportion of code generations that pass all test cases (7.7 test cases on average per problem) for at least k samples. See the benchmarks introduction for how scoring works across benchmarks.What does the n argument control?
n is the number of code generation samples produced for each task, used by the pass@k metric. It is set to 200 by default, meaning your LLM is invoked 200 times on the same prompt per task unless you lower it.What is the difference between n and k?
n is the total number of samples generated per task, while k is the number of top samples chosen for the pass@k metric. You supply k when calling evaluate(), for example benchmark.evaluate(model=gpt_4, k=10).How do I run HumanEval on a custom LLM?
generate_samples method on your DeepEvalBaseLLM subclass, then create a benchmark with HumanEval(tasks=[...], n=100) and call benchmark.evaluate(model=gpt_4, k=10). See the benchmarking guide for using any custom LLM.