Agents' Last Exam (ALE)

What is Agents' Last Exam?

Agents' Last Exam (ALE) tests whether an AI agent can finish a long piece of paid professional work in real software and hand back a file that a program can check. The agent gets a task description, some input files and a sandbox with the target application already installed. It might need to set up a toolpath in Siemens NX, build a scene in Unreal Engine, run a mold-flow study in Moldex3D or clean up a brain scan in FSLeyes. Then it writes its deliverable to an output folder, and a grader compares that file against a hidden reference.

UC Berkeley RDI built it. The leaderboard page says the project is "co-led by UC Berkeley RDI and the RDI Foundation," and Snorkel AI hosts a mirror of the board. The paper, led by Yiyou Sun with 310 listed authors, went up on June 3, 2026, with a v2 on June 11. It reports 1,490 task instances drawn from 960 unique workflows, sorted into 13 industry clusters and 55 subfields that map to the US O*NET / SOC 2018 occupational taxonomy. The tasks come from practitioners' completed projects, and between 250+ and 300+ experts took part. The paper abstract says 250+, and the leaderboard page and the paper body say 300+.

The paper places ALE against three groups of earlier benchmarks. Knowledge tests measure what a model knows, not what it can do. Agentic suites such as OSWorld and WebArena cover narrow domains with tasks curators wrote themselves. Economic evaluations such as GDPval and RLI use real work but cover 16 and 14 of the 55 industries, and they need human graders. ALE covers all 55 subfields and grades with code. In the open-sourced workflows, 93.2% use deterministic judges and 6.8% use an LLM judge.

The scope is software-mediated work. The abstract calls it "non-physical industries," and the domains run through manufacturing, biomolecular design, animation, robotics, agriculture, finance, law and education. The leaderboard lists After Effects, Siemens NX, Unreal Engine, Moldex3D, Rhino 3D and FSLeyes "and 49 more applications."

ALE is not a GUI-grounding test and not a test of how agents cope with OS popups or a drifting desktop. That is OSWorld 2 and ScreenSpot-Pro territory. ALE grades the work product.

How the tasks and harness work

Every task follows the same directory contract, per the repo README and the paper. input/ holds read-only files, software/ holds the pre-installed application, output/ is the only folder the agent can write to, and reference/ holds the hidden ground truth. The agent sees a natural-language description and a deliverable spec. The rubric and the reference stay out of reach.

Tasks run in Linux or Windows VMs and containers. The supported sandbox providers include Google Cloud, AWS, local Docker and QEMU/KVM. Agent frameworks run as shipped: Codex, Claude Code, Cursor CLI, OpenClaw, ALE-Claw, Droid, Gemini CLI, Grok CLI, Hermes, ForgeCode, Terminus, OpenHands and others. Two open harnesses ship with the framework, the official Claude Code CLI and the in-tree OpenClaw harness. ALE-Claw is the reference harness.

The paper also adds a mode it calls GUI-as-Tool. It exposes 14 desktop actions over an MCP bridge next to ordinary shell commands, so one model can reason over shell output and screenshots in the same loop. This matters because 34% of public instances name graphical software as the primary tool. A CLI-only agent cannot do those tasks the intended way.

Each run has a five-hour wall-clock cap. When the cap hits, the harness stops the agent and the grader scores whatever is in output/. The paper states no step limit. In the paper's main runs, 3.8% of runs hit the cap, about 1% for lightweight harnesses and 5.7% for OpenClaw. Capped runs averaged a score of 20.7 against 33.2 for runs that finished earlier.

The paper's cost estimate for a single run is "$3-10 on average," taking "tens of minutes to hours" per task. That puts one ALE task in a different weight class from a coding benchmark item.

How scoring works

Graders are deterministic wherever the output allows it: exact or hash matches, tolerance checks on numeric and tabular fields, geometric distance, world-state checks, and narrow yes/no probes from a vision model. Free-text rubrics appear only when nothing deterministic fits. The leaderboard sums it up as "Hidden references plus deterministic graders, not LLM-as-a-judge."

Scores combine a gate and a score. Each task has binary gates, such as "the file parses" or "the toolpath is collision-free." Fail a gate and the task scores zero, however good the rest of the work looks. Pass the gates and the graded checks add up to partial credit.

The board reports two numbers per row:

• Pass Rate is the share of tasks fully completed. It is the headline metric.
• Score is the average graded outcome, partial credit included.

The two numbers sit far apart. The Overall leader posts a 38.2% pass rate and a 63.2 score. An agent that gets most of a deliverable right still fails the task if one required piece is missing, and Pass Rate is the number to read if your workflow has no use for a nearly finished file.

The paper reports three-run standard deviations of roughly ±1 to ±3 on score for a subset of configurations. Most configurations have one run.

Versions, tiers and tabs

The live board is labeled ALE-V1, snapshot dated September 29, 2026. The paper's main runs come from the GPT-5.5, Claude Opus 4.7 and Claude Fable 5 generation. The board adds GPT-5.6, GPT-6, Claude Opus 5 and 5.5, Kimi K3, Grok 4.5 and Muse Spark 1.3.

About 150 tasks are public. The sources give three counts: the Snorkel page says 147, the repo README says "around 150," and the paper says both 150 public instances and "the full 152-task public set." In the paper's split, 1,017 instances are private and 323 were pending quality control. Held-out private tasks score the official leaderboard, and the public pool rotates. Snorkel's page says: "Every ~6 months, a new public subset releases with fresh instances. Private tasks rotate into the public pool, retired public tasks rotate out."

The public set is harder than the full pool. The paper ran Claude Code with Opus 4.7 on everything and got a higher pass rate on the full pool, because the public set holds the entire hardest tier while the private pool leans toward easier tasks. Per-cluster pass rates still track closely between the two, at a Pearson r of 0.89.

The paper defines three difficulty tiers plus a CLI subset:

The board has five tabs: Overall plus those four. Each tab has its own rows and picks its own best effort setting per model. The board does not say how many tasks sit behind each tab or whether they match the paper's counts. Most Near-Term pass rates fit a count out of 67, but two Codex GPT-5.5 values do not, so this page does no per-task division. Costs and runtimes below are the board's published totals for each tab.

Current leaderboard

Every row below comes from the Snorkel mirror, whose embedded data names the canonical board as its source, with an as-of date of 2026-09-29 and version ALE-V1. The canonical page loads its rows by script, so none were read from it directly. The board publishes no per-row evaluation date and no field saying who ran each row, whether a vendor submitted it, or whether anyone verified it. That holds for every row here, including rows where the harness vendor and the model vendor are the same company. No lab system card or release post retrieved for this page reports an ALE score.

Three more things to know before reading the numbers:

• Costs drift. The canonical site recalculates cost estimates without rerunning evaluations. GPT-5.6 Sol's Overall cost moved from $762 to $772, and GPT-6 Astra's cost and token counts were republished on September 29 with pass rate, score and runtime unchanged. The board does not publish how it estimates cost.
• Claude Fable 5 rows carry a flag. The source site states that "the variant served during evaluation may differ from the model's full-capability tier, and re-runs cannot guarantee the higher-tier variant is selected. Scores may understate the model's true ceiling."
• Runtime changed method. The source site updated its runtime methodology on July 10, 2026, so runtimes do not line up with earlier snapshots.

Cost and runtime are totals for the tab's task set. Effort is the board's best-effort pick, meaning the setting with the highest pass rate, ties broken by score.

On the Overall tab, Claude Code with Claude Opus 5.5 at Max effort leads at 38.2% pass and a 63.2 score. Codex with GPT-6 Astra at Max is second at 34.2% and 59.3, 4.0 points behind on pass rate. Those are model-plus-harness pairs. The board has no Opus 5.5 run in Codex and no Astra run in Claude Code, so the gap between the two models alone is unmeasured.

Overall tab, Codex harness

Same harness and tab for every row, best-effort rows only. Comparable within this table. Comparing to the Claude Code table compares model-plus-harness pairs, not models.

Astra leads the Codex field by 2.0 points over GPT-6 Sol and Muse Spark 1.3. Sol gets there for $318 against Astra's $1,100. Meta's Muse Spark 1.3 is the only non-OpenAI model in this table, and it ties Sol on pass rate with the shortest runtime of the top four.

Overall tab, Claude Code harness

Same harness and tab for every row, best-effort rows only. Comparable within this table. Not comparable model-to-model with the Codex table.

Opus 5.5 is 6.0 points clear of Opus 5 here. Below them, Qwen3 8-Max, Kimi K3 and Opus 4.8 tie at 27.0%, and the first two get there for about an eighth of Opus 4.8's $3,985.

The chart uses the Near-Term tab because it is the tier where today's agents finish about half the work, so the spread between rows means something. Four rows sit on the cost frontier: GPT-5.6 Luna at $69 and 49.3%, GPT-6 Sol at $81 and 50.7%, GPT-6 Astra at $283 and 52.2%, and Opus 5.5 at $480 and 53.7%. GPT-6 Sol gets within 1.5 points of Astra for 29% of the cost. Opus 5.5 adds another 1.5 points over Astra for 1.7x the money. Opus 5, GPT-5.6 Sol, Muse Spark 1.3 and GPT-5.6 Terra all sit behind the frontier.

Effort ladders in the Overall tab

Same model, harness and tab within each block; compare across effort settings only. Source is the Snorkel mirror's per-row effort data alone.

More test-time compute helps Astra and stops helping the other two. Astra's Max beats its Low by 4.6 points for 2.0x the cost. GPT-6 Sol peaks at XHigh, and its Max costs $110 more for 0.6 fewer points. Opus 5 peaks at High. Its Max and XHigh both cost more and pass fewer tasks.

Other harnesses on the Overall tab

One row per harness, each with a different model; this is a reference list, not a ranking. Rows do not rank against each other or against the Codex and Claude Code tables.

The board has no Gemini 4 entry, and no source retrieved for this page reports a Gemini 4 ALE result.

Same model, different harness

Rows share a model and the High effort setting; only the harness differs. Comparable within each model's block. The Codex GPT-5.5 High row is not listed because the board's best-effort row for that pair is XHigh.

Opus 4.7 spans 7.6 points across five harnesses, from 13.5% in Droid to 21.1% in Cursor CLI. Its own vendor's harness, Claude Code, comes second to last. GPT-5.5 spans 3.3 points, from 19.7% in Droid to 23.0% in ALE-Claw. The agent architecture wrapped around the model is part of the result.

Difficulty tiers

Each tab is a different task set, and each cell is that tab's best-effort row, so effort can change across a row. Compare within a column only. Cells read pass / score / estimated tab cost.

Claude Code with Opus 5.5 is the top row on every tab. Codex with Astra is second on Overall, Near-Term and Full-Spectrum and ties Muse Spark 1.3 at 33.3% on ALE-CLI.

The Last-Exam column needs a different reading. Its pass rates are task counts out of 38, so 15.8% is six tasks, 13.2% is five, 10.5% is four and 2.6% is one. A single task moves the number 2.6 points. Opus 5.5's lead there is one task over GPT-6 Sol and Opus 5, and two over Astra. Other Last-Exam rows: Qwen3 8-Max in Claude Code at XHigh 10.5 / 24.1 / $217, Kimi Code with Kimi K3 10.5 / 20.6 / $308, Grok Build with Grok 4.5 10.5 / 23.7 / $115, and Claude Code with Kimi K3 7.9 / 20.5 / $200.

Historical results

ALE is four months old, so its history is one model generation inside a few harnesses. The paper's June runs and the September board are two vintages with different setups, and they stay in separate tables.

Paper runs, June 2026

From arXiv 2606.05405 Table 1, v2 dated June 11, 2026. Every harness had the GUI-as-Tool mode added. Full-pass rates; these are the paper authors' runs on older models and do not compare directly with the live board. The paper's per-row effort settings were not extracted for this page.

The paper also held the harness fixed and swapped the model. In OpenClaw, 14 models ran from GPT-5.5 at 21.1% down to Grok 4.3 at 4.3%, with Opus 4.7 at 15.1, Gemini 3.1 Pro and Opus 4.6 at 14.1, DeepSeek V4 Pro at 12.4 and Sonnet 4.6 at 9.9. Appendix D.4 puts the effect of the model at roughly 3x the spread from harness choice among well-built systems. On the paper's ALE-CLI subset, Codex with GPT-5.5 scored 23.3%, and Claude Code and ForgeCode with Sonnet 4.6 scored 16.7% and 13.3%. The live ALE-CLI tab is a different run set.

Board series, September 2026

Each series fixes one harness and one effort setting. Different vintage and setup from the paper table; the board makes no GUI-as-Tool statement.

Two facts stand out. Claude Code with Opus 4.7 scored 13.2% in the paper and 13.8% on the board at High, so the two run sets agree within 0.6 points on the one configuration they share. And within Claude Code at High effort, Opus 4.7 to Opus 5 is an 18.4-point jump. The hardest tier moved too. The paper abstract says the "average full pass rate is below 1%" on Last-Exam across mainstream configurations, and its best was one task of 38. The board's best is now six.

Where agents fail

The paper analyzed failed Claude Code runs with Opus 4.7. About three quarters of failures were understanding and approach problems: the agent misread what the deliverable needed or picked the wrong way to make it. About a quarter were execution and skill problems. Appendix D.4 traces the main bottleneck to reasoning and domain knowledge. One habit shows up again and again. Agents write ad-hoc scripts instead of opening the domain software the task was built around.

GUI use is the clearest gap. A third of public tasks name a graphical application as the primary tool, and in most configurations the share of GUI actions stays small. Snorkel's reading-group post puts GUI evaluation at about 50x slower than CLI.

Results also vary by field. Averaged over harnesses for Fable 5 and GPT-5.5, computational mathematics and agriculture and environment score highest, about 55 to 85%. Business and legal land around 50 to 55%, and education is lowest, below 25%.

Spending more does not buy a better result. In Claude Code, Fable 5 spends $4,340 for a 48.7 score while Opus 5.5 spends $1,340 for 63.2. In the paper's cost analysis, ALE-Claw with GPT-5.5 reached a 45.8 mean score for $326, while ALE-Claw with Opus 4.7 cost 3.6x more for 40.5. Claude Code with Fable 5 spent $2,402, the most of any configuration, for the same 40.5. Cursor with Composer 2.5 was the cheapest, at 38.5 for $87. ALE-Claw with GPT-5.5 also finished in about 51 hours of wall-clock time against 466 hours for Claude Code with Opus 4.8.

Timeouts cost real points. Capped runs average 20.7 against 33.2 for others, and the Last-Exam tier times out at 6.4%, double the 3.1% and 3.2% of the easier tiers.

Contamination defense rests on the private pool and the six-month rotation. The roughly 150 public tasks are readable by anyone, training pipelines included. Neither the paper nor the leaderboard documents a contamination measurement.

Several things are not published: the cost estimation method, per-row evaluation dates, who ran each board row, the number of tasks behind the Overall tab, a step limit, and any vendor-reported ALE score. OpenAI's GPT-6 Astra model page and Anthropic's Opus 5.5 announcement list pricing and dates but no ALE result.

How ALE compares to related benchmarks

GDPval is the closest relative. Both grade professional deliverables, but GDPval relies on human expert graders, and the ALE paper counts its coverage at 16 of 55 industries against ALE's 55. ALE swaps the human grader for a deterministic checker, which makes it cheaper to rerun and harder to argue with. The cost is that it only works where a checker can be written.

Terminal and coding benchmarks look easy next to it. The ALE paper says Codex with GPT-5.5 "achieves 82% on Terminal-Bench," yet the same pair reached 23.3% on ALE-CLI and stayed under 50% on ALE's easiest tier. The paper does not name a Terminal-Bench version, so do not read that 82% against Terminal-Bench 4.0. SWE-bench is the test-graded coding analogue. It grades by execution too, but for one profession instead of 55.

OSWorld 2 tests GUI-only computer use on general desktop apps. ALE gives the agent a shell and a GUI and grades the professional file it produces, not the clicks. GAIA asks for short answers to general-assistant questions, while ALE wants multi-hour deliverables. For narrower workflow suites, see AutomationBench for business automation and the Harvey legal agent benchmark for law.

What it means for teams choosing a model

Pick the harness and the model together. On the same model at High effort, the harness moved Opus 4.7 by 7.6 points and GPT-5.5 by 3.3. The top two Overall rows, Claude Code with Opus 5.5 and Codex with Astra, have never run in each other's harness. Test the model inside the agent framework you will ship.

If you run Codex, GPT-6 Sol is the value pick. At XHigh it ties Astra's XHigh pass rate, 32.2%, for $318 against $898. Astra's Max buys 2.6 more points than Sol's Max for 2.6x the cost. On Near-Term, with each at its best effort, Sol reaches 50.7% for $81 against Astra's 52.2% for $283.

If you run Claude Code, move from Opus 5 to Opus 5.5. At Max effort, Opus 5.5 passes 7.3 points more for 0.90x the cost. Fable 5 at XHigh spends $4,340 for 48.7, and Opus 5.5 at Max scores 14.5 points higher for 31% of that, with the caveat that the efforts differ and the board flags the Fable 5 variant.

Do not default to maximum effort. Of the three models with a published effort ladder, only Astra does best at Max. Opus 5 does best at High, and GPT-6 Sol at XHigh.

Plan for failure on hard work. The best Overall configuration still fails 61.8% of tasks, every Codex configuration fails at least 65.8%, and the best Last-Exam result is six tasks of 38. The board gives no signal on which tasks will fail, so any production use needs a verification step that checks the deliverable before anyone relies on it. A frontier model on ALE is a strong drafter for professional files, not a finisher.

The LLM leaderboard compares models across benchmarks. If your work runs through CAD, simulation or other engineering software, the more useful number is a pass rate on your own deliverables; the engineering page covers how to set up that kind of evaluation in Klu.