SWE-bench Benchmark

What is the SWE-bench Benchmark?

The SWE-bench benchmark is an end-to-end evaluation framework designed to assess the software engineering capabilities of large language models (LLMs) and autonomous AI coding agents. Unlike traditional coding benchmarks that evaluate models on their ability to write isolated functions or solve algorithmic puzzles (like HumanEval or MBPP), SWE-bench challenges AI agents to resolve real-world, complex issues from popular open-source Python repositories on GitHub.

Introduced by researchers from Princeton University and the University of Chicago, SWE-bench shifts the paradigm of AI evaluation from passive code generation to active problem-solving within established codebases. Each task in the benchmark corresponds to an actual merged pull request that resolved a reported bug or implemented a new feature. The AI agent is given a description of the issue, typically the original GitHub issue text, and complete access to the repository's source code exactly as it existed before the issue was fixed. The agent must then navigate the codebase, understand the relationships between different files and modules, work out the necessary changes, and generate a patch that addresses the issue without breaking existing functionality.

This evaluation approach demands a broad set of software engineering skills. Agents must excel at information retrieval to locate the relevant files, demonstrate contextual understanding of long-form, complex codebases, plan multi-step modifications, and execute precise edits. Because the benchmark relies on real issues, the complexity and ambiguity of the tasks mirror the genuine challenges faced by human software engineers on a daily basis.

Benchmark Variants: SWE-bench Full, SWE-bench Lite, and SWE-bench Verified

To accommodate different evaluation needs, resource constraints, and the fast-moving capabilities of AI models, the SWE-bench framework is distributed across three primary variants: SWE-bench Full, SWE-bench Lite, and SWE-bench Verified.

SWE-bench Full represents the original, comprehensive dataset. It consists of 2,294 task instances drawn from 12 popular open-source Python repositories, such as Django, SymPy, scikit-learn, and matplotlib. This variant is the hardest test of an agent's capabilities, requiring it to navigate vast codebases and solve complex, often open-ended problems. Evaluating models on the Full dataset is computationally expensive and time-consuming, since it requires setting up and running thousands of containerized environments.

SWE-bench Lite was introduced to provide a more accessible and faster evaluation pathway. It is a curated subset of 300 tasks sampled from the Full dataset. The tasks selected for the Lite variant were chosen to be more self-contained, meaning the solutions typically require modifying fewer files and rely less on deep, repository-wide architectural knowledge. SWE-bench Lite serves as a practical testing ground for rapid prototyping, smaller models, or researchers with limited compute budgets, offering a proxy for overall software engineering performance.

SWE-bench Verified is the variant that matters most for frontier model comparisons. Developed with OpenAI in 2024, it consists of 500 tasks from the Full dataset that have been manually vetted by human annotators. This verification process ensures that the issue descriptions are clear, the unit tests are deterministic, and the tasks are unambiguously solvable. By removing poorly specified issues and flaky tests, SWE-bench Verified provides a more stable and trusted signal of a model's coding proficiency, reducing the noise inherent in raw GitHub data.

The official site has since added two further variants: SWE-bench Multimodal, which draws issues from JavaScript repositories where understanding the bug requires reading screenshots or rendered output, and SWE-bench Multilingual, which extends execution-based evaluation beyond Python. The maintainers also run a "bash-only" track on the Verified set, where every model is evaluated with the same minimal mini-SWE-agent scaffold, a plain ReAct loop with nothing but a bash shell, so model scores can be compared without scaffold differences confounding them.

Evaluation Methodology: Containerized Execution-Based Testing

The distinguishing feature of SWE-bench is its execution-based evaluation methodology, which judges correctness by running actual unit tests rather than relying on static analysis, fuzzy matching, or LLM-as-a-judge heuristics.

When an AI agent proposes a patch for a specific task, the SWE-bench evaluation harness automatically spins up a Docker container that replicates the exact environment, dependencies, and configuration of the target repository at the time the issue was reported. The harness applies the agent's proposed patch to the codebase and then executes two distinct suites of unit tests: the "Fail-to-Pass" suite and the "Pass-to-Pass" suite.

The Fail-to-Pass tests are the unit tests that were specifically added or modified by the human developer who originally resolved the issue. In the codebase's pre-patch state, these tests fail because the bug is present or the feature is missing. To succeed on the SWE-bench task, the agent's proposed patch must cause all of these tests to pass, confirming that the agent has implemented the required fix or feature.

Simultaneously, the harness runs the Pass-to-Pass tests. These are existing, historical unit tests in the repository that were already passing before the issue was addressed. The agent's patch must not cause any of these tests to fail. This step verifies that the agent has not introduced new bugs, broken backward compatibility, or caused regressions in unrelated parts of the codebase.

An AI agent is only considered to have successfully resolved a SWE-bench task if it achieves a perfect score on both suites: every Fail-to-Pass test must pass, and every Pass-to-Pass test must remain passing. This binary, execution-driven scoring mechanism gives an objective, ground-truth measure of software engineering competence.

Frontier Agent Leaderboard & Historical Progression

The SWE-bench leaderboard has become a closely watched signal of state-of-the-art capability in autonomous software engineering. Progress has been fast, driven by advances in foundational models, reasoning frameworks, and multi-agent architectures.

Early zero-shot evaluations of models like GPT-4 on SWE-bench Full resulted in solve rates below 2%, showing how difficult the benchmark is. The introduction of agentic workflows, such as Devin by Cognition, SWE-agent by Princeton researchers, and OpenHands (formerly OpenDevin), then improved performance substantially by letting models iteratively search, edit, and test their code within an isolated environment.

The table below shows the historical progression of top-performing models and agent frameworks on SWE-bench. Scores are only comparable within the same benchmark variant (Full, Lite, or Verified), and vendor-reported scores use each vendor's own scaffold, which typically runs a few points above the standardized bash-only track. Figures through November 2025 are vendor-reported; later rows come from the official swebench.com leaderboard, checked August 2026.

Two things stand out in the 2026 data. First, the Verified leaderboard has flattened just under 80 percent: the top all-agents entries as of August 2026 (live-SWE-agent and Sonar Foundation Agent, both driving Claude Opus 4.5) sit at 79.2 percent, barely above scaffolded results from late 2025. The remaining fifth of the benchmark concentrates its hardest, most ambiguous tasks, and progress there is slow. Second, on the bash-only track, models a fraction of frontier price (Gemini 3 Flash at 75.8 percent, MiniMax M2.5 at 75.8 percent, DeepSeek-V3.2 at 70.0 percent) now land within a few points of the best frontier models, which has shifted competition toward cost per resolved issue rather than raw solve rate.

Common Failure Modes of Software Engineering Agents on SWE-bench

Despite rapid progress, even state-of-the-art software engineering agents exhibit consistent failure modes when evaluated on SWE-bench. Understanding these limitations matters for developers building enterprise AI coding tools.

One of the most frequent failure modes is context window mismanagement. Real-world repositories contain hundreds of files and tens of thousands of lines of code. Agents often struggle to prioritize the correct files for retrieval, leading to either missing necessary context (failing to find the bug) or overflowing their context windows with irrelevant information (diluting the model's focus).

Cascading errors in multi-step planning represent another significant hurdle. Agents frequently propose a plausible initial fix but fail to anticipate its downstream effects on other modules. When execution-based testing reveals these regressions, the agents often become trapped in cyclical debugging loops, applying patch after patch without addressing the root cause, ultimately exhausting their token limits or execution time.

Agents also struggle with implicit architectural assumptions. Many SWE-bench issues require an understanding of unspoken design patterns or framework-specific conventions (e.g., Django's ORM behavior). Models trained primarily on disparate code snippets often fail to grasp these holistic, repository-wide invariants, resulting in patches that solve the immediate logic error but violate the broader architectural intent.

Practical Implications for Enterprise Coding Agent Evaluation

For enterprise organizations adopting AI coding assistants, SWE-bench carries practical implications. It shows that strong performance on standard coding interviews (like LeetCode) does not translate directly to productivity in mature enterprise codebases.

When evaluating vendor solutions or open-source agents, enterprises should prioritize SWE-bench Verified scores over basic code generation metrics. An agent that scores highly on SWE-bench has demonstrated the ability to read existing code, respect legacy constraints, and debug autonomously, skills that are more valuable in an enterprise setting than generating boilerplate from scratch.

Furthermore, the benchmark highlights that integration matters as much as raw intelligence. The top scores on SWE-bench are consistently achieved by agentic frameworks (combining LLMs with bash, python, and search tools) rather than raw models. Enterprises must ensure their AI coding tools have secure, high-bandwidth access to their internal repositories, linters, and CI/CD pipelines to get real value from autonomous agents.

Frequently Asked Questions

Why does SWE-bench only use Python repositories? The original dataset used Python because of its large open-source ecosystem, straightforward dependency management, and standardized testing frameworks like pytest. That limitation no longer holds for the benchmark family: the official SWE-bench Multimodal variant draws tasks from JavaScript repositories with visual bug reports, and SWE-bench Multilingual extends execution-based evaluation to additional languages. The headline Verified numbers, however, still refer to the Python-only 500-task set.

Is it possible for models to memorize the SWE-bench dataset? Data contamination is a valid concern, since the GitHub issues and pull requests used in SWE-bench are publicly available and may have been included in the training data of foundational models. However, researchers have found that merely memorizing the issue text is insufficient to achieve high scores, as the agent must dynamically apply the exact patch in the correct environment. SWE-bench also periodically updates its dataset with newer issues to mitigate contamination risks.

How does SWE-bench Verified differ from the original dataset? SWE-bench Verified is a 500-task subset of the original Full dataset that was manually audited by human engineers to ensure high quality. It removes tasks with ambiguous instructions, poorly written issue descriptions, or flaky unit tests that pass or fail non-deterministically. This makes it a more reliable and fair evaluation target for measuring genuine model improvements.

Can I run SWE-bench locally on my own machine? Yes, the SWE-bench evaluation harness is fully open-source and can be run locally. However, it requires a significant amount of disk space, compute power, and a robust Docker installation, as it must build and execute hundreds of complex containerized environments. Many developers use cloud infrastructure to run the benchmark efficiently.