HumanEval Benchmark

What is the HumanEval Benchmark?

OpenAI developed HumanEval and released it in 2021 with the Codex model. Codex, a GPT model fine-tuned on GitHub code, powers GitHub Copilot and solves 28.8% of the original evaluation problems, outperforming GPT-3 and GPT-J. With repeat sampling, its success rate increases to 70.2%. However, it struggled with complex docstrings and variable bindings, raising concerns about safety, security, and economic impacts.

The HumanEval benchmark is a dataset designed to evaluate the code generation capabilities of large language models (LLMs). It consists of 164 hand-crafted programming challenges, each including a function signature, docstring, body, and several unit tests, averaging 7.7 tests per problem. These challenges assess a model's understanding of language, algorithms, and simple mathematics, and are comparable to simple software interview questions.

HumanEval is used to measure the functional correctness of code generated by LLMs from docstrings. The models are evaluated based on their ability to generate code that passes the provided unit tests. The pass@k metric is used to assess the performance, where 'k' different solutions are generated by the model, and if any of these solutions pass all the unit tests, the model is considered to have solved the problem.

This benchmark was created to prevent data leakage by ensuring the problems were not included in the training sets of code generation models, thus providing a fair assessment of a model's ability to generate novel code. The HumanEval dataset has become a popular tool for assessing the progress and capabilities of code generation models since its inception in mid-2021.

The benchmark is also used to compare the performance of different models. In the archived 2024 comparison below, Language Agent Tree Search with GPT-4 was the leading reported system. Despite its popularity, it is recognized that solving programming questions in HumanEval does not encompass all aspects of a code model's potential applications, such as code explanation, docstring generation, code infilling, and writing tests. Therefore, while HumanEval is a significant step towards evaluating code generation models, it is part of an ongoing effort to improve code benchmarking.

Historical scaffolded-system leaderboard (June 25, 2024)

In this June 25, 2024 snapshot, GPT-4o with LDB assistant led the reported scaffolded systems. Claude 3 Opus, released on March 4, 2024, scored 84.9 in the same archived table.

The leaderboard for the HumanEval benchmark ranks models based on the innovative pass@1, pass@10, and pass@100 metrics. These metrics offer a more meaningful and practical assessment of a model's ability to solve problems.

Additional protocol context for the historical agent and scaffold snapshot

The table above preserves the agent and scaffold comparison published on this page as of June 25, 2024 and retained in the repository archive dated July 17, 2024. These are system-level HumanEval pass@1 results: the named base model was embedded in a debugging, search, repair, planning, or multi-agent scaffold. They measure the combined system rather than a single direct model call.

The archived page did not identify a versioned leaderboard export, exact base-model snapshots, prompts, decoding settings, attempt budgets, scaffold configurations, execution harnesses, or whether every row used identical HumanEval tests and pass@1 estimation. Those differences can materially change a score. The results therefore cannot be ranked directly against the December 2024 EvalPlus table, whose entries are direct-model evaluations and whose HumanEval+ column uses an expanded test suite.

Manual maintenance should trace each row to its primary paper and implementation, record immutable model and scaffold versions, verify the HumanEval harness and sampling budget, and rerun comparable systems under one documented protocol before refreshing this historical snapshot.

EvalPlus Direct-Model Snapshot (December 26, 2024)

Additive direct-model results under a separate protocol.

The latest official EvalPlus leaderboard data remains the results snapshot last modified on December 26, 2024. It does not include models released in 2026. The table reports direct-model pass@1 results rather than agent or repair scaffolds, keeping the model comparison separate from systems such as LDB, AgentCoder, or search-based code agents.

HumanEval+ applies the EvalPlus test suite, which adds substantially more tests to the original HumanEval tasks and often exposes fragile solutions that pass the original tests. These direct-model scores should not be compared as if they used the same protocol as multi-step agent, debugging, search, or repair scaffolds.

How is the HumanEval benchmark used in code generation research?

The HumanEval benchmark evaluates the functional correctness of code generated by large language models (LLMs) through 164 programming challenges. These challenges test a model's grasp of language, algorithms, and mathematics by requiring it to generate code from docstrings that must pass specific unit tests. The pass@k metric assesses model performance, considering a model successful if any of its 'k' generated solutions pass all tests.

This benchmark not only supported historical model and system rankings, including the archived GPT-4-based leaders above, but also highlights biases and evaluation flaws, such as a focus on a narrow range of programming concepts and an abundance of simpler questions. While HumanEval is crucial for evaluating code generation, it doesn't cover all potential applications like code explanation or docstring generation, indicating the need for broader benchmarking efforts.

Additional current context: The historical leading-model claim above belongs to the archived leaderboard snapshot. Direct-model and scaffolded-system results should be evaluated under their separately documented protocols.

HumanEval's scope extends beyond Python, with versions available for other programming languages, enhancing its utility in code generation research.

How does HumanEval work?

HumanEval is a benchmark dataset designed to evaluate the functional correctness of code generated by large language models (LLMs). It consists of 164 hand-crafted programming challenges, each including a function signature, docstring, body, and several unit tests, with an average of 7.7 tests per problem.

The evaluation process involves generating multiple solutions based on a given prompt and running unit tests on these solutions. If any of the generated solutions pass the unit tests, it's counted as a win. This approach is encapsulated in the pass@k metric, where k represents the number of different solutions generated by the model. For instance, pass@1 means the model's first solution is tested, pass@10 means any of the model's first 10 solutions can pass the tests, and so on.

While HumanEval has become a popular benchmark for code generation models, it's important to note that it doesn't capture all aspects of real-world usage of code models. For example, tasks like code explanation, docstring generation, code infilling, and writing tests are not evaluated by HumanEval.

The HumanEval benchmark was introduced in the paper "Evaluating Large Language Models Trained on Code" by OpenAI, and the evaluation harness for the dataset is available on GitHub. The leaderboard for the benchmark, hosted by Papers with Code, ranks models based on the pass@k metrics.

Additional current harness context: EvalPlus extends that protocol with additional tests and publishes both original HumanEval and HumanEval+ pass@1 results. Because evaluation prompts, decoding settings, and scaffolds affect scores, comparisons should use the same harness and report those settings alongside the model version.

The Pass@k Metric

The pass@k metric is a performance evaluation measure used primarily for models that generate code, such as OpenAI's Codex. It assesses the probability that at least one of the top k-generated code samples for a given problem passes the unit tests. This metric is particularly useful because it aligns more closely with the practices of human developers, focusing on functional correctness rather than text similarity.

When evaluating pass@k, a dataset of natural language/code pairs is used. For each natural language prompt, the model generates k code snippets. If at least one of these code snippets is correct, the model is considered to have succeeded for that prompt. The pass@k value is then the fraction of prompts for which the model succeeded.

The pass@k metric can be calculated using a numerically stable method, as shown in the provided Python function from one of the search results. This method is important because it avoids numerical instability or overflow that can occur when dealing with large values of n, c, and k, where n is the total number of samples, c is the number of correct samples, and k is the number of top samples considered.

It's important to note that the pass@k metric does not behave like a simple probability because the trials are not independent. The generation of code snippets is based on a stochastic procedure, which means that the probability of success does not scale linearly with the number of attempts (k). Therefore, pass@100 will not be 99.999999% even if pass@1 is 28%, because the trials are not independent and the probability of success does not compound in a straightforward manner.

When to use HumanEval

The HumanEval benchmark is used in several scenarios within the field of machine learning, particularly when dealing with large language models (LLMs) and code generation:

1. Evaluating Functional Correctness: HumanEval is primarily used to measure the functional correctness of code generated by LLMs. It provides a set of programming challenges that the models must solve by generating code from docstrings. The generated code is then evaluated based on its ability to pass the provided unit tests.

2. Comparing Model Performance: HumanEval serves as a benchmark for comparing the performance of different LLMs in code generation tasks. It provides a standardized set of challenges that all models must solve, allowing for a fair comparison of their capabilities.

3. Research and Development: HumanEval is used in research to study the capabilities and limitations of LLMs in code generation. It helps expose biases and shortcomings in code generation models, providing insights that can guide the development of improved models.

4. Benchmarking New Models: When new models are developed, HumanEval can be used to benchmark their performance against existing models. This helps determine whether the new models offer any improvements over the current state-of-the-art.

5. Developing New Evaluation Metrics: The use of HumanEval has led to the development of new evaluation metrics like the pass@k metric, which provides a more meaningful and practical assessment of a model's ability to solve programming challenges.

However, it's important to note that while HumanEval is a valuable tool, it does not capture all aspects of a code model's potential applications, such as code explanation, docstring generation, code infilling, and writing tests. Therefore, it should be used in conjunction with other evaluation methods to get a comprehensive understanding of a model's capabilities.

Limitations of HumanEval

The HumanEval benchmark, while a significant tool for evaluating code generation models, has several limitations:

1. Bias Towards Certain Concepts: HumanEval has been found to have a significant bias towards a limited number of programming concepts, with many concepts not represented at all. This could lead to an overestimation of a model's performance on code generation tasks.

2. Limited Scope: The benchmark consists of problems that are mostly algorithmic and similar to interview-style coding questions, which may not reflect the complexity of real-world software development tasks. For instance, in a corporate setting, developers often work with multiple files and existing codebases, which is not captured by the single-function problems in HumanEval.

3. Weak Unit Tests: The unit tests provided with HumanEval problems are sometimes too weak, allowing incorrect implementations to pass. This undermines the reliability of the benchmark in assessing the correctness of generated code.

4. Training Data Contamination: Since HumanEval has been around for a while, there is a risk that the models have been exposed to the problems or similar ones during training, leading to potential data leakage and invalidating the evaluation results.

5. Narrow Evaluation Metric: The pass@k metric, while useful, provides a narrow view of a model's capabilities. It does not account for other important aspects of code generation such as code explanation, docstring generation, code infilling, handling Stack Overflow questions, and writing tests.

6. Popularity and Memorization: The popularity of HumanEval might lead to models being fine-tuned specifically to perform well on its problems, which does not necessarily translate to general code generation ability. This can also lead to memorization of solutions rather than genuine problem-solving.

7. Lack of Real-world Relevance: The benchmark does not fully capture the nuances and expectations of real-world code model usage, as it is based on a limited set of standalone functions rather than complex, multi-file projects.

8. Non-representative of Diverse Programming Tasks: The tasks in HumanEval may not represent the diversity of programming tasks found in different domains and real-world scenarios.

Researchers and practitioners in the field are aware of these limitations and there is ongoing work to improve code benchmarking to address these issues.