BigCodeBench: A New Benchmark for Evaluating LLMs on Programming Tasks

What is BigCodeBench?

BigCodeBench is a new benchmark designed to evaluate large language models (LLMs) on solving practical and challenging programming tasks. It was created to address the limitations of existing benchmarks like HumanEval, which have been criticized for being too simple and not representative of real-world programming tasks.

Key features of BigCodeBench include:

• 1,140 function-level tasks that challenge LLMs to follow instructions and compose multiple function calls from 139 libraries
• An average of 5.6 test cases per task with 99% branch coverage
• Complex, user-oriented instructions for each task, including functionality descriptions, input/output formats, and error handling
• Verified interactive examples for each task
• Tasks that require more complex reasoning and problem-solving skills compared to other benchmarks

Historical full-set leaderboard snapshot (June 28, 2024)

In the June 28, 2024 full-set snapshot, GPT-4o-2024-05-13 led with a Complete score of 61.1 and an Instruct score of 51.1.

The BigCodeBench leaderboard ranks models based on their performance in two main scenarios: BigCodeBench-Complete and BigCodeBench-Instruct. The benchmark uses the Pass@1 metric with greedy decoding to assess LLM performance, measuring the percentage of tasks correctly solved with the first generated code snippet via curated test cases. Additionally, an Elo rating system is used to provide a comprehensive ranking of the models.

This is a historical BigCodeBench leaderboard snapshot of the full 1,140-task benchmark published on June 28, 2024; it is not a current ranking.

BigCodeBench-Hard leaderboard snapshot (April 16, 2025)

BigCodeBench-Hard is a more challenging, user-facing subset of roughly 150 tasks. It uses a different task set and evaluation configurations from the full benchmark above, so their scores are not directly comparable and the lower Hard-set numbers do not show that model performance regressed.

The latest official published results remain the April 16, 2025 BigCodeBench-Hard data; the project has not published a newer snapshot. The official leaderboard ranks its default Hard view by the average of calibrated Pass@1 on BigCodeBench-Complete and BigCodeBench-Instruct, evaluated with greedy decoding.

Model settings also matter within the Hard snapshot: reasoning effort, output limits, chat formatting, and response prefilling can change the result.

In this Hard-set snapshot, the highest average calibrated Pass@1 is 35.80%, achieved by Claude-3.7-Sonnet-20250219 with its reported reasoning configuration. Among the five leading Hard averages above, DeepSeek-R1 has the highest Complete score, while o3-mini at medium reasoning has the highest Instruct score. Reports should identify whether they use the Hard or full set and preserve the model's decoding and prompting settings.

How does BigCodeBench work?

BigCodeBench evaluates LLMs in two main scenarios:

• BigCodeBench-Complete — LLMs are required to finish the implementation of a function based on detailed instructions in the docstring.

• BigCodeBench-Instruct — A more challenging variant designed to evaluate instruction-tuned LLMs, where requirements are described in a more conversational and less verbose manner.

The benchmark uses the Pass@1 metric with greedy decoding to assess LLM performance. This measures the percentage of tasks correctly solved with the first generated code snippet via curated test cases.

How were the tasks created?

The tasks in BigCodeBench were created through a systematic "Human-LLM collaboration process":

1. Started with ODEX as a "seed dataset" of short human intents and Python one-liners from Stack Overflow
2. Used GPT-4 to expand these one-liners into comprehensive function-level tasks
3. 20 human experts with extensive Python experience guided GPT-4 to refine tasks and add test cases
4. Tasks and test cases were examined in a local environment, pre-evaluated on other LLMs, and cross-checked by additional human experts
5. 11 human experts solved a sample of tasks, achieving an average human performance of 97%

How do LLMs perform on BigCodeBench?

Performance on BigCodeBench is significantly lower than human performance:

• The best model (GPT-4o) achieves a calibrated Pass@1 of 61.1% on BigCodeBench-Complete and 51.1% on BigCodeBench-Instruct
• There is a notable performance gap between closed and open LLMs
• 149 tasks in BigCodeBench-Complete and 278 tasks in BigCodeBench-Instruct remain unsolved by all models
• Only 6 tasks in BigCodeBench-Complete and 14 tasks in BigCodeBench-Instruct are fully solved by all models

An Elo rating system is used to rank models, with GPT-4o outperforming other models by a large margin.

How can I evaluate my model on BigCodeBench?

BigCodeBench is a sophisticated benchmark that rigorously evaluates the code generation capabilities of large language models (LLMs) in realistic scenarios. It surpasses traditional HumanEval-like tasks by incorporating complex instructions and diverse function calls, offering a more comprehensive assessment of LLMs' programming abilities.

The bigcodebench Python package comprises three key components:

1. A curated dataset of challenging programming tasks
2. Robust scripts for generating code samples
3. Advanced evaluation tools for assessing model performance

Leveraging the EvalPlus framework, BigCodeBench provides a flexible and extensible platform for evaluating code generation tasks. This architecture enables researchers and developers to conduct thorough, comparative analyses of various LLMs' performance in practical programming contexts.

BigCodeBench provides a simple evaluation framework accessible via PyPI:

1. Install the package:

   pip install bigcodebench --upgrade

2. Generate code samples:

   bigcodebench.generate --model [model_name] --subset [complete|instruct] ...

3. Post-process the generated code:

   bigcodebench.sanitize --samples samples.jsonl --calibrate

4. Evaluate the code (preferably using Docker):

   docker run -v $(pwd):/app bigcodebench/bigcodebench-evaluate:latest --subset [complete|instruct] --samples samples-sanitized-calibrated

Future Developments

The BigCodeBench team has outlined several areas for future improvement:

1. Multilingualism: Extending beyond Python to other programming languages
2. Rigorousness: Improving test case coverage and assessment accuracy
3. Generalization: Including tasks with emerging libraries like transformers and langchain
4. Evolution: Addressing the challenge of evolving libraries and potential test set contamination
5. Interaction: Exploring LLMs as Agents in less constrained sandbox environments

BigCodeBench represents a significant step forward in evaluating the programming capabilities of LLMs, providing a more comprehensive and challenging benchmark that better reflects real-world programming tasks.