What does CritPt v2 Hard measure?
CritPt tests research-level physics problem solving. Artificial Analysis evaluates the 40-task hard subset of the 70-task CritPt v2 benchmark. Models can use Python and code execution, and the official grader checks their answers. The published metric is pass@1 over five repeats. These settings come from Artificial Analysis's intelligence benchmarking methodology.
Current results
This October 28, 2026 snapshot keeps every published model configuration separate. Missing scores are omitted. Cost and time columns, when reported, belong to this benchmark in the same model observation; they are not costs for the complete Intelligence Index.
| Model configuration | Score | Cost per task (USD) | Time per task (seconds) | Observation and source |
|---|---|---|---|---|
| GPT-6 Astra (High) | 71.5% | $0.7142 | 181.1 | 2026-10-28; AA Index v5.0 |
| GPT-6 Astra (Medium) | 71.0% | $0.4067 | 105.4 | 2026-10-28; AA Index v5.0 |
| GPT-6 Astra (Max) | 70.5% | $1.1377 | 249.8 | 2026-10-28; AA Index v5.0 |
| GPT-6 Astra (Xhigh) | 70.5% | $0.8493 | 197.9 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Sol (Max) | 68.5% | $0.2300 | 238.5 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Sol (Xhigh) | 68.0% | $0.1671 | 194.3 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Sol (High) | 67.5% | $0.1345 | 152.6 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Sol (Medium) | 65.0% | $0.0714 | 94.9 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Terra (Max) | 64.5% | $0.2505 | 420.9 | 2026-10-28; AA Index v5.0 |
| Claude Opus 5.5 (Xhigh, Default Fallback) | 63.0% | $0.9657 | 276.0 | 2026-10-28; AA Index v5.0 |
| Claude Opus 5.5 (Max, Default Fallback) | 61.0% | $2.5767 | 634.9 | 2026-10-28; AA Index v5.0 |
| Claude Fable 5.1 (High, Default Fallback) | 60.5% | $2.3876 | 380.4 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Terra (High) | 60.5% | $0.0691 | 115.0 | 2026-10-28; AA Index v5.0 |
| GPT-6 Astra (Low) | 60.0% | $0.2367 | 73.1 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Sol (Low) | 59.5% | $0.0433 | 57.9 | 2026-10-28; AA Index v5.0 |
| Claude Opus 5.5 (High, Default Fallback) | 58.0% | $0.3844 | 111.1 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Terra (Xhigh) | 57.5% | $0.1177 | 196.9 | 2026-10-28; AA Index v5.0 |
| Claude Fable 5.1 (Medium, Default Fallback) | 56.5% | $1.3829 | 225.5 | 2026-10-28; AA Index v5.0 |
| Claude Fable 5.1 (Xhigh, Default Fallback) | 56.5% | $4.3322 | 660.1 | 2026-10-28; AA Index v5.0 |
| Claude Sonnet 5.5 (Max, Default Fallback) | 56.0% | $1.9267 | 654.7 | 2026-10-28; AA Index v5.0 |
| GPT-5.6 Sol (Max) | 56.0% | — | — | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Terra (Medium) | 56.0% | $0.0405 | 66.2 | 2026-10-28; AA Index v5.0 |
| Claude Sonnet 5.5 (Xhigh, Default Fallback) | 54.0% | $0.4478 | 195.9 | 2026-10-28; AA Index v5.0 |
| Muse Wonder Full (Max) | 53.0% | — | — | 2026-10-28; AA Index v5.0 |
| Claude Opus 5.5 (Medium, Default Fallback) | 52.5% | $0.2512 | 72.2 | 2026-10-28; AA Index v5.0 |
| Gemini 4 Argon (High) | 52.5% | $1.0743 | 839.9 | 2026-10-28; AA Index v5.0 |
| Claude Fable 5.1 (Max, Default Fallback) | 51.0% | $4.5529 | 712.7 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Terra (Low) | 48.5% | $0.0178 | 27.0 | 2026-10-28; AA Index v5.0 |
| Muse Wonder (Max) | 48.0% | $1.8136 | 323.2 | 2026-10-28; AA Index v5.0 |
| Claude Fable 5.1 (Low, Default Fallback) | 47.5% | $0.8923 | 147.7 | 2026-10-28; AA Index v5.0 |
| Claude Sonnet 5.5 (High, Default Fallback) | 47.5% | $0.1805 | 77.7 | 2026-10-28; AA Index v5.0 |
| Muse Spark 1.3 (Max) | 46.0% | $0.6570 | 438.3 | 2026-10-28; AA Index v5.0 |
| Kimi K3.1 Pro | 45.0% | $0.7017 | 995.9 | 2026-10-28; AA Index v5.0 |
| Claude Haiku 5.5 (Xhigh) | 44.5% | $0.0594 | 279.0 | 2026-10-28; AA Index v5.0 |
| Gemini 3.8 Flash (Medium) | 44.0% | $0.8267 | — | 2026-10-28; AA Index v5.0 |
| Gemini 3.8 Flash (High) | 43.5% | $1.1535 | 486.9 | 2026-10-28; AA Index v5.0 |
| Claude Haiku 5.5 (High) | 42.5% | $0.0334 | 163.1 | 2026-10-28; AA Index v5.0 |
| GPT-6 Luna (Max) | 41.5% | $0.0325 | 386.9 | 2026-10-28; AA Index v5.0 |
| Claude Haiku 5.5 (Max) | 41.0% | $0.1013 | 388.5 | 2026-10-28; AA Index v5.0 |
| Grok 4.7 (Low) | 41.0% | $0.7943 | 570.2 | 2026-10-28; AA Index v5.0 |
| Kimi K3 (Max) | 39.5% | — | — | 2026-10-28; AA Index v5.0 |
| Ling 3.1 Flash 1019 | 37.5% | $0.3203 | — | 2026-10-28; AA Index v5.0 |
| GPT-6 Luna (Xhigh) | 35.5% | $0.0129 | 159.8 | 2026-10-28; AA Index v5.0 |
| Grok 4.7 (High) | 35.0% | $1.8941 | 976.6 | 2026-10-28; AA Index v5.0 |
| MiMo-V2.6-Pro | 34.0% | $0.1652 | 1756.2 | 2026-10-28; AA Index v5.0 |
| Claude Opus 5.5 (Low, Default Fallback) | 33.5% | $0.1191 | 33.8 | 2026-10-28; AA Index v5.0 |
| GLM-5.3 (Max) | 33.5% | $1.5426 | 1421.9 | 2026-10-28; AA Index v5.0 |
| GPT-6.1 Terra (Non-reasoning) | 32.5% | $0.0170 | 22.7 | 2026-10-28; AA Index v5.0 |
| Step 5 Preview | 32.5% | $0.3584 | 350.4 | 2026-10-28; AA Index v5.0 |
| Grok 4.7 (Xhigh) | 32.0% | $2.2369 | 1145.2 | 2026-10-28; AA Index v5.0 |
| Gemini 3.8 Flash (Low) | 31.5% | — | — | 2026-10-28; AA Index v5.0 |
| MiniMax-M3.1-Flash | 31.0% | $0.1665 | 1206.0 | 2026-10-28; AA Index v5.0 |
| GPT-6 Luna (High) | 30.5% | $0.0096 | 115.3 | 2026-10-28; AA Index v5.0 |
| Claude Haiku 5.5 (Medium) | 30.0% | $0.0167 | 88.5 | 2026-10-28; AA Index v5.0 |
| MiMo-V2.6-Flash | 29.5% | $0.0616 | 2189.3 | 2026-10-28; AA Index v5.0 |
| Claude Sonnet 5.5 (Medium, Default Fallback) | 28.5% | $0.0926 | 39.1 | 2026-10-28; AA Index v5.0 |
| Qwen3.8 Max (0902) | 28.5% | $1.7571 | 2381.8 | 2026-10-28; AA Index v5.0 |
| Muse Spark Open (Xhigh) | 27.5% | $0.5069 | 349.5 | 2026-10-28; AA Index v5.0 |
| Beam 1.0 | 26.5% | $0.0887 | 77.4 | 2026-10-28; AA Index v5.0 |
| MiniMax-M3 | 26.5% | $0.3349 | 911.6 | 2026-10-28; AA Index v5.0 |
| GPT-6 Luna (Medium) | 26.0% | $0.0055 | — | 2026-10-28; AA Index v5.0 |
| DeepSeek V4.1 Flash 1017 (Max) | 24.0% | $0.2000 | 449.1 | 2026-10-28; AA Index v5.0 |
| Mistral Large 4 Preview | 24.0% | $1.0169 | 781.5 | 2026-10-28; AA Index v5.0 |
| DeepSeek V4 Pro 0813 (Max) | 23.5% | $0.8902 | 1117.7 | 2026-10-28; AA Index v5.0 |
| Claude Sonnet 5.5 (Low, Default Fallback) | 22.5% | $0.0731 | 29.0 | 2026-10-28; AA Index v5.0 |
| Claude Haiku 5.5 (Low) | 21.5% | $0.0083 | 42.7 | 2026-10-28; AA Index v5.0 |
| GLM-5.3 (Low) | 21.0% | $0.7577 | 849.2 | 2026-10-28; AA Index v5.0 |
| Inkling Small | 18.5% | $0.1362 | 267.6 | 2026-10-28; AA Index v5.0 |
| Inkling (Xhigh) | 18.0% | $0.5517 | 330.3 | 2026-10-28; AA Index v5.0 |
| Qwen3.8 27B (Medium) | 16.5% | — | — | 2026-10-28; AA Index v5.0 |
| Nemotron 3 Ultra 550B A55B (Reasoning) | 15.0% | $0.3574 | 346.1 | 2026-10-28; AA Index v5.0 |
| Nemotron 3.5 Super VL | 13.5% | $0.1318 | 203.9 | 2026-10-28; AA Index v5.0 |
| GPT-5 (High) | 11.5% | — | — | 2026-10-28; AA Index v5.0 |
| K2 Horizon 375B A23B | 11.5% | $0.0000 | 83.1 | 2026-10-28; AA Index v5.0 |
| GPT-6 Luna (Low) | 9.5% | $0.0019 | 23.3 | 2026-10-28; AA Index v5.0 |
| Muse Glimmer (High) | 7.0% | $0.0684 | 291.3 | 2026-10-28; AA Index v5.0 |
| Kimi K3 (Low) | 5.5% | — | — | 2026-10-28; AA Index v5.0 |
| Qwen3.8 27B (Low) | 5.5% | — | — | 2026-10-28; AA Index v5.0 |
| GPT-6 Luna (Non-reasoning) | 4.5% | $0.0011 | 12.2 | 2026-10-28; AA Index v5.0 |
| Qwen3.8 27B (Xhigh) | 4.0% | $0.5249 | 1421.3 | 2026-10-28; AA Index v5.0 |
| Gemini 3.5 Flash-Lite | 3.5% | $0.1828 | 65.9 | 2026-10-28; AA Index v5.0 |
| Command A+ | 2.5% | $0.0000 | 354.9 | 2026-10-28; AA Index v5.0 |
| Qwen3.8 27B (Non-reasoning) | 2.0% | — | — | 2026-10-28; AA Index v5.0 |
| Granite 4.2 8B | 0.0% | — | — | 2026-10-28; AA Index v5.0 |
| Mistral Small 4 (Reasoning) | 0.0% | — | — | 2026-10-28; AA Index v5.0 |
| NVIDIA Nemotron 3 Nano 4B | 0.0% | — | — | 2026-10-28; AA Index v5.0 |
How to compare these scores
Compare rows within this version and subset. A score from the full 70-task set, an earlier CritPt version, or a different tool and grading setup cannot establish improvement or regression against this hard-subset board. The benchmark score is also distinct from the aggregate AA Intelligence Index and the Klu Index.
For broader scientific workflows that require producing files in a terminal, see Terminal-Bench Science. For academic question answering, see HLE-Diamond.