ScreenSpot-Pro

What is ScreenSpot-Pro?

ScreenSpot-Pro is a GUI grounding benchmark, a narrow computer vision test for screen agents. The model gets a full-screen screenshot of a professional desktop application and a short natural-language instruction. It has to return the location of the one UI element that satisfies the instruction. Nothing else is asked. There is no plan to write, no task to finish, no environment to drive. The benchmark checks whether the model can find a small icon on a 5K screen and point at it.

Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang and Tat-Seng Chua built it across the National University of Singapore, East China Normal University and Hong Kong Baptist University. They released the dataset and paper on 2025-01-04 and posted it to arXiv on 2025-04-04.

The reason it exists is that the earlier ScreenSpot benchmark had become too easy to be useful. ScreenSpot cropped screenshots down to local regions of consumer mobile, web and light desktop apps, and the average target covered 2.01% of the image. ScreenSpot-Pro keeps the whole screen and moves to IDEs, CAD tools, creative suites, scientific packages, office software and operating system settings. The average target now covers 0.07% of the screenshot, about one twenty-ninth of the old size.

That change broke the models of the day. At release the best of ten models tested, OS-Atlas-7B, scored 18.9%. GPT-4o scored 0.8%. By October 2026 the published scores sit in the 70s and 80s, and the highest of them all wrap the model in cropping, zooming or multi-step search. Anthropic, Google DeepMind, Qwen, H Company and OpenAI all cite it among their LLM benchmarks, and the gap between a model's raw score and its wrapped score is the most useful thing it measures.

How the task works

The dataset holds 1,581 instructions, each on its own screenshot. They come from 23 applications across five industries and three operating systems: Windows, macOS and Linux. The paper groups the applications into six genres.

• Development and programming. VS Code, PyCharm, Android Studio, Quartus, VMware Fusion.
• Creative. Photoshop, Premiere, Illustrator, Blender, FruitLoops Studio, Unreal Engine, DaVinci Resolve.
• CAD and engineering. AutoCAD, SolidWorks, Inventor, Vivado.
• Scientific and analytical. MATLAB, Origin, Stata, EViews.
• Office suite. Word, PowerPoint, Excel.
• Operating system commons. Windows 11, macOS Sonoma, Ubuntu 24.04.

Annotators were experts with at least five years in each application. They recorded their normal workflow with a silent screenshot tool on monitors above 1080p with display scaling off, then drew the target box themselves. Resolutions in the set include 2560x1440, 3840x2160, 5120x2880, 2560x1664 and 2880x1800. At least two annotators reviewed every instance. Ambiguous instructions were rewritten until exactly one target existed, and boxes cover only the clickable region of the element.

Targets split 62.6% text and 37.4% icon. An element counts as an icon only when it has no text label, so a labeled toolbar button counts as text. The icon share is where models fail, as the failure modes section shows.

Inputs, output and scoring

The input is the screenshot plus the instruction. The output is a single click coordinate. Models that emit a bounding box have the center of that box scored as the click. A prediction is correct when the point lands inside the ground-truth box. The model gets no accessibility tree and no DOM, only pixels.

The headline number is accuracy over all 1,581 instructions. The official grounding leaderboard reports the micro-average under greedy decoding, with breakdowns by application, by genre and by text versus icon. The maintainers collect submissions directly rather than running models themselves.

Each item is static. One screenshot in, one prediction out, with no episode, no sandbox, no step limit and no time limit. The reference evaluation scripts are on GitHub under the MIT license, and the dataset is on Hugging Face. A full run is 1,581 model calls without a wrapper, small enough to rerun on every model release.

How the labs run it

"One screenshot, one click" is the benchmark's definition. Frontier labs rarely report it that way. Each lab that publishes a number uses its own setup, and most add a tool or a loop around the model.

• Anthropic reports Claude in the Opus 4.8 System Card (section 8.12.5) with adaptive thinking at maximum effort, in two modes. One gives the model no tools. The other gives it Python tools. Each score is a five-run average.
• Google DeepMind runs Gemini 3 Pro with a "capture screenshot" function-calling tool that passes the screenshot back to the model, at media_resolution "extra_high", per its evaluation methodology PDF. DeepMind ran the non-Gemini models itself through provider APIs because no self-reported or leaderboard numbers existed.
• Qwen gives Qwen3-VL a computer_use tool limited to left_click and mouse_move, tells it to center the cursor tip on the target, and lets it give up if it cannot find the element (Qwen3-VL Technical Report, Appendix B.11).
• H Company reports Holo2 both single-step and with "agentic localization," which refines the guess for up to three steps over the 4K screenshot (Holo2 release post).
• OpenAI says GPT-6 Astra "achieves state-of-the-art results on Agents' Last Exam, AutomationBench, and ScreenSpot Pro" in its announcement. It has published no score, effort setting, scaffold or resolution. The GPT-6 Astra System Card and the GPT-6.1 Sol addendum have no ScreenSpot-Pro section.

Image resolution also changes scores. Anthropic's Opus 4.7 release raised image input to 2,576 pixels on the long edge, about 3.75 megapixels and more than three times the limit of earlier Claude models. ScreenSpot-Pro screenshots run up to 5120x2880, so even that limit means a 5K screen is downscaled before the model sees it.

Versions

ScreenSpot-Pro has no numbered versions. The only dated maintainer updates are in the repo README.

• 2025-01-04. Paper and dataset released.
• 2025-02-21. The README lists projects already using it as a benchmark, including OmniParser v2, Qwen2.5-VL, UI-TARS, UGround and AGUVIS.
• 2025-05-19. SE-GUI results added, 47.2% for the 7B model and 35.9% for the 3B, trained on 3,000 open-source samples.

Every task also has a Chinese instruction, translated with GPT-4 and then reviewed. That set ships as ScreenSpot-Pro-CN, described in paper Appendix A and run with the repo script run_ss_pro_cn.sh. The leaderboard page was last updated 2026-09-10.

Current leaderboard

Every score on this page is vendor-reported or leaderboard-submitted. No independent run of a frontier model on ScreenSpot-Pro exists, and OpenAI has published no number.

The highest published score is Claude Opus 4.8 at 87.9% with Python tools, under Anthropic's harness. The top of the official leaderboard is GroundCascade at 85.1%, a cascade over two open models. Those two numbers come from different harnesses and do not rank against each other. The tables below are grouped so that rows inside each table do compare.

Anthropic harness

One harness, so the four rows compare directly. Do not compare them to the leaderboard or DeepMind tables.

Opus 4.8 beats Opus 4.7 by 2.8 points without tools and by 0.3 points with them. The tools close almost the entire gap between the two models.

Google DeepMind's Gemini 3 Pro table

DeepMind ran every row. The Gemini rows compare with each other. The non-Gemini rows use settings DeepMind did not document, so they work only as a rough reference to older models, not as a comparison with the Anthropic table.

The PDF says of extra_high, "This setting is coming soon to our API." The 72.7% figure used a setting customers could not yet select when it was published.

Official leaderboard, wrapped entries

Greedy decoding, micro-average, leaderboard-submitted. Each entry adds a submitter-built wrapper over open models. These rows compare with the single-pass leaderboard table only as the wrapper-versus-base pairs called out below.

The leaderboard JSON holds 99 entries. None of the entry names contains Opus, Sonnet, Gemini or GPT-6. The top of the official board is open 8B to 32B models with wrappers, not frontier APIs.

Official leaderboard, single-pass entries

Same protocol, one forward pass per example, no wrapper. Legacy rows predate current models and used their own prompts, so the bottom of this table is history more than competition.

The three frontier-API rows near the bottom tell you nothing about current Claude or GPT models. Nobody knows who ran them, when, or with what prompt.

Holo2, single step versus agentic localization

One harness from H Company, so rows compare directly. All eight rows also appear on the official leaderboard.

The 8B model with three steps of refinement scores 71.4%, ahead of the 235B model in a single step at 70.6%. The 8B and 4B models gain 12.5 and 11.4 points from the loop. The 235B gains 7.9.

Qwen3-VL

Qwen's own harness, with the click-only computer_use tool. It does not compare with the other tables, and Qwen's own Table 2 shows "-" for Gemini 2.5 Pro, GPT-5 and Claude Opus 4.1.

The Thinking variants score at or below Instruct at every size, by 0.2 to 3.2 points. Under Qwen's harness, extra reasoning bought nothing on grounding.

Historical progression

Each row carries its own harness. Read the column as a timeline of what was published, not as one model improving on one setup.

The most telling row is from April 2025. GPT-4o scored 0.8% when asked to click directly. Used as the planner in ScreenSeekeR, telling OS-Atlas-7B where to look, it lifted that model from 18.9% to 48.1%. A model that cannot click can still search well.

Failure modes and limitations

Icons. At release OS-Atlas-7B scored 28.1% on text targets and 4.0% on icons. The paper blames professional toolbars with many functions, apps that assume the user already knows them, and icons with domain meanings that rarely show up in web training data.

Target size. The average target covers 0.07% of the screen. The paper's Figure 2 shows accuracy falling as box area shrinks for SeeClick, OS-Atlas, UGround and Qwen2-VL on ScreenSpot-v2. ScreenSpot-Pro's average target is about one twenty-ninth the size of ScreenSpot's.

Resolution and crop settings outweigh model choice. Gemini 3 Pro moves 12.2 points between the "high" and "extra_high" media resolution settings. In the paper's Table 5, ReGround with OS-Atlas-7B scores 25.1% at 512x512 crops, 34.2% at 768, 40.2% at 1024 and 40.1% at 1280, while UGround peaks at 768 with 28.8% and falls to 26.3% at 1280. There is no single best crop size across models. On the leaderboard, GPT5-high (resized) scored 6.0% against 18.5% for GPT5-minimal (resized).

Every wrapper helps. Every model reported both with and without a wrapper gains from it. Anthropic's Python tools add 5.6 points to Opus 4.8 and 8.1 to Opus 4.7. Leaderboard wrappers add 5.6 to 9.6 points over the same weights. Holo2's agentic loop adds 7.9 to 12.5. A score without a stated wrapper is not usable for comparison.

Grounding is not task completion. The paper's Limitations section says the benchmark leaves out agent planning and execution of the kind OSWorld tests, partly to avoid legal risk from software licensing. A model that clicks the right button still has to know which button comes next.

Small per-app counts. With 1,581 items across 23 apps, per-app scores rest on thin samples. Stata has 49 targets and EViews has 50. Read per-application breakdowns as directional.

Contamination is unchecked. The dataset has been public on Hugging Face and GitHub since 2025-01-04. None of the Anthropic system card, the DeepMind methodology, the Qwen3-VL report or the Holo2 post reports a contamination or decontamination check for ScreenSpot-Pro.

Reporting gaps. Anthropic gives its 95% confidence intervals only in a figure. OpenAI has published no number, effort, scaffold or run count. DeepMind documents its screenshot tool only for Gemini 3. No lab publishes cost or latency for the task, which is why this page has no cost or time chart.

How ScreenSpot-Pro compares to related benchmarks

ScreenSpot is the predecessor, with cropped screenshots from mobile, web and light desktop apps. Its average target is 2.01% of the image against 0.07% here. Treat ScreenSpot scores as a separate benchmark, not an easier subset.

OSWorld runs 361 tasks end to end with an agent in a live Ubuntu VM. ScreenSpot-Pro isolates one step of that loop, putting the cursor on the right pixel, and drops planning and execution. Anthropic reports Opus 4.8 at 83.4% on OSWorld-Verified in the same system card. If your agent fails tasks, ScreenSpot-Pro tells you whether perception is the cause. OSWorld tells you whether the whole agent works.

MMMU tests image understanding with text answers and no pixel output. Answering a question about an image in words is a different skill from returning the coordinate of a toolbar button, so a high MMMU score does not predict ScreenSpot-Pro. When you shortlist large multimodal models for a screen agent, look at ScreenSpot-Pro instead.

AutomationBench measures enterprise workflow automation through apps and APIs with no screenshot grounding at all. OpenAI's Astra claim pairs ScreenSpot-Pro with AutomationBench and Agents' Last Exam.

What ScreenSpot-Pro means for teams choosing a model

Each point below stays within one harness.

1. Turn on a crop or zoom tool before comparing models. Under Anthropic's harness, Python tools take Opus 4.8 from 82.3% to 87.9% and Opus 4.7 from 79.5% to 87.6%. Comparing Claude versions without tools measures a setup you should not ship.
2. Do not upgrade for grounding alone if you already crop. Opus 4.8 gains 2.8 points over Opus 4.7 without tools and 0.3 points with them. If your agent already zooms, a 0.3-point gain will not show up in your traffic.
3. Check which resolution setting a Gemini score used. Gemini 3 Pro scores 72.7% at extra_high and 60.5% at high. When DeepMind published the PDF, extra_high was not yet available in the API, so the number you could reproduce was the lower one.
4. Budget calls per screenshot, not calls per task. The top leaderboard entry, GroundCascade at 85.1%, makes about 2.6 model calls per screenshot. Zoom, cascade and MVP wrappers lift open 8B to 32B models by 5.6 to 9.6 points over the same weights. For a high-volume agent, an open grounding model with a wrapper is a serious option against a frontier API.
5. Small models with a refinement loop match large ones without it. Holo2-8B with three-step agentic localization scores 71.4%, level with Holo2-235B single-step at 70.6%. If your latency budget allows three passes, the 8B model reaches the 235B model's single-step accuracy.
6. Send screenshots at native resolution where the model allows it. Opus 4.7 accepts images up to 2,576 pixels on the long edge, and ScreenSpot-Pro screens go up to 5120x2880. Check what your pipeline does to 4K and 5K captures before blaming the model.
7. Ignore claims without numbers. OpenAI says GPT-6 Astra is state-of-the-art on ScreenSpot Pro and has published nothing to check. Run your own screenshots through each candidate with the same crop tool and the same test-time compute budget.

The public numbers mostly measure the wrapper, so the test that counts is your own. Capture screens from the applications your agent drives, label the targets, and score each candidate under the harness you will deploy. Klu runs that kind of evaluation on your own workflow, as described for operations teams, and the LLM leaderboard tracks public results as new frontier models ship.