What does CritPt v2 Hard measure?
CritPt tests research-level physics problem solving. Artificial Analysis evaluates the 40-task hard subset of the 70-task CritPt v2 benchmark. Models can use Python and code execution, and the official grader checks their answers. The published metric is pass@1 over five repeats. These settings come from Artificial Analysis's intelligence benchmarking methodology.
Current results
This current snapshot keeps every published model configuration separate. Missing scores are omitted. Cost and time columns, when reported, belong to this benchmark in the same model observation; they are not costs for the complete Intelligence Index.
| Model configuration | Score | Cost per task (USD) | Time per task (seconds) | Observation and source |
|---|---|---|---|---|
| Claude Fable 5.2 (Max, Default Fallback) | 76.5% | $6.3530 | 1225.6 | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.2 (Xhigh, Default Fallback) | 73.5% | $2.4721 | 500.9 | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.2 (High, Default Fallback) | 73.0% | $1.0494 | 206.2 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Astra (Medium) | 72.0% | $0.4067 | 161.0 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Astra (High) | 71.5% | $0.7142 | 276.8 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Astra (Xhigh) | 71.0% | $0.8493 | 367.6 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Astra (Max) | 70.5% | $1.1377 | 509.8 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Sol (Max) | 69.5% | $0.2300 | 391.9 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Sol (Xhigh) | 69.5% | $0.1671 | 288.0 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Carbon (High) | 69.0% | $2.4380 | 1992.1 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Carbon (Medium) | 69.0% | $1.9299 | 1668.4 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Sol (High) | 68.5% | $0.1345 | 225.1 | 2026-11-14; AA Index v5.0.1 |
| Claude Opus 5.5 (Max, Default Fallback) | 68.0% | $2.5767 | 854.9 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Terra (Max) | 68.0% | $0.2505 | 564.7 | 2026-11-14; AA Index v5.0.1 |
| Claude Opus 5.5 (Xhigh, Default Fallback) | 67.5% | $0.9657 | 332.0 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Sol (Medium) | 67.0% | $0.0714 | 112.6 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Sol (Max) | 66.5% | $0.3840 | 249.3 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Hydrogen (Xhigh) | 66.0% | $1.8995 | 980.6 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Carbon (Low) | 65.0% | $1.9123 | 1951.9 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Astra (Low) | 63.0% | $0.2367 | 90.7 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Sol (Low) | 62.5% | $0.0433 | 68.4 | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.1 (High, Default Fallback) | 62.0% | $2.3876 | 445.3 | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.2 (Medium, Default Fallback) | 61.5% | $0.3669 | 75.4 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Hydrogen (High) | 61.5% | $0.8777 | 480.9 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Terra (High) | 61.5% | $0.0691 | 166.1 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Hydrogen (Medium) | 61.0% | $0.3744 | 211.9 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Terra (Xhigh) | 60.5% | $0.1177 | 276.0 | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.1 (Xhigh, Default Fallback) | 60.0% | $4.3322 | 767.0 | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.2 (Low, Default Fallback) | 60.0% | $0.3130 | 58.5 | 2026-11-14; AA Index v5.0.1 |
| GPT-5.6 Sol (Max) | 60.0% | — | — | 2026-11-14; AA Index v5.0.1 |
| Claude Opus 5.5 (High, Default Fallback) | 59.5% | $0.3844 | 135.8 | 2026-11-14; AA Index v5.0.1 |
| Claude Sonnet 5.5 (Max, Default Fallback) | 59.0% | $1.9267 | 1024.2 | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.1 (Medium, Default Fallback) | 58.5% | $1.3829 | 257.5 | 2026-11-14; AA Index v5.0.1 |
| Claude Sonnet 5.5 (Xhigh, Default Fallback) | 57.5% | $0.4478 | 244.4 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Terra (Medium) | 56.5% | $0.0405 | 98.5 | 2026-11-14; AA Index v5.0.1 |
| Muse Wonder Full (Max)Muse Code subscription | 56.5% | — | — | 2026-11-05; AA Index v5.0.1 |
| Claude Opus 5.5 (Medium, Default Fallback) | 54.0% | $0.2512 | 89.7 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Argon (Medium) | 53.5% | $0.7824 | — | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.1 (Max, Default Fallback) | 52.5% | $4.5529 | 808.9 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Argon (High) | 52.5% | $1.1744 | 1100.9 | 2026-11-14; AA Index v5.0.1 |
| Claude Sonnet 5.5 (High, Default Fallback) | 51.5% | $0.1805 | 94.6 | 2026-11-14; AA Index v5.0.1 |
| Muse Wonder (High)Meta Model API and Muse Code | 51.0% | $1.1682 | 422.7 | 2026-11-14; AA Index v5.0.1 |
| GPT-5.6 Terra (Max) | 50.5% | $0.6236 | 385.2 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Terra (Low) | 50.5% | $0.0178 | 40.4 | 2026-11-14; AA Index v5.0.1 |
| Muse Wonder (Max)Meta Model API and Muse Code | 50.5% | $1.8136 | 604.0 | 2026-11-14; AA Index v5.0.1 |
| Claude Opus 5 (Max) | 50.0% | $2.9047 | 996.5 | 2026-11-14; AA Index v5.0.1 |
| Claude Fable 5.1 (Low, Default Fallback) | 49.5% | $0.8923 | 162.1 | 2026-11-14; AA Index v5.0.1 |
| Kimi K3.1 Pro | 49.5% | $0.7017 | 1582.5 | 2026-11-14; AA Index v5.0.1 |
| Gemini 4 Hydrogen (Low) | 48.0% | $0.1225 | 71.4 | 2026-11-14; AA Index v5.0.1 |
| Muse Spark 1.3 (Max) | 48.0% | $0.6570 | 676.0 | 2026-11-14; AA Index v5.0.1 |
| Claude Haiku 5.5 (Xhigh) | 47.5% | $0.0594 | 372.9 | 2026-11-14; AA Index v5.0.1 |
| Gemini 3.8 Flash (High) | 46.5% | $1.1535 | 588.8 | 2026-11-14; AA Index v5.0.1 |
| Claude Haiku 5.5 (High) | 45.0% | $0.0334 | 213.2 | 2026-11-14; AA Index v5.0.1 |
| Gemini 3.8 Flash (Medium) | 45.0% | $0.8267 | 458.6 | 2026-11-14; AA Index v5.0.1 |
| Claude Haiku 5.5 (Max) | 43.5% | $0.1013 | 586.5 | 2026-11-14; AA Index v5.0.1 |
| Grok 4.7 (Low) | 43.5% | $0.7943 | 682.5 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Luna (Max) | 42.5% | $0.0325 | 464.8 | 2026-11-14; AA Index v5.0.1 |
| Kimi K3 (Max) | 40.5% | — | — | 2026-11-14; AA Index v5.0.1 |
| Grok 4.6 (Xhigh) | 40.0% | $0.8930 | 879.0 | 2026-11-14; AA Index v5.0.1 |
| Muse Ember (Max) | 39.0% | $0.0538 | 184.0 | 2026-11-14; AA Index v5.0.1 |
| Tencent Hy4 | 39.0% | $0.3076 | 867.5 | 2026-11-14; AA Index v5.0.1 |
| Ling 3.1 Flash 1019 | 38.5% | $0.3203 | — | 2026-11-14; AA Index v5.0.1 |
| Muse Ember (Xhigh) | 38.5% | $0.0405 | 139.3 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Luna (Xhigh) | 37.0% | $0.0129 | 199.5 | 2026-11-14; AA Index v5.0.1 |
| Grok 4.7 (High) | 37.0% | $1.8941 | 1363.5 | 2026-11-14; AA Index v5.0.1 |
| Claude Opus 5.5 (Low, Default Fallback) | 36.0% | $0.1191 | 41.8 | 2026-11-14; AA Index v5.0.1 |
| Inkling 2 Small | 35.0% | $0.2487 | 1783.7 | 2026-11-14; AA Index v5.0.1 |
| MiMo-V2.6-Pro | 34.0% | $0.1652 | 2074.1 | 2026-11-14; AA Index v5.0.1 |
| Claude Sonnet 5.5 (Medium, Default Fallback) | 33.5% | $0.0926 | 51.7 | 2026-11-14; AA Index v5.0.1 |
| Gemini 3.8 Flash (Low) | 33.5% | — | — | 2026-11-14; AA Index v5.0.1 |
| GLM-5.3 (Max) | 33.5% | $1.5426 | 2133.0 | 2026-11-14; AA Index v5.0.1 |
| GPT-6.1 Terra (Non-reasoning) | 33.5% | $0.0170 | 32.2 | 2026-11-14; AA Index v5.0.1 |
| Grok 4.7 (Xhigh) | 33.5% | $2.2369 | 1656.7 | 2026-11-14; AA Index v5.0.1 |
| MiniMax-M3.1-Flash | 32.5% | $0.1665 | 1185.2 | 2026-11-14; AA Index v5.0.1 |
| Step 5 Preview | 32.5% | $0.3584 | 527.4 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Luna (High) | 31.0% | $0.0096 | 148.6 | 2026-11-14; AA Index v5.0.1 |
| Claude Haiku 5.5 (Medium) | 30.5% | $0.0167 | 111.4 | 2026-11-14; AA Index v5.0.1 |
| GPT-5.6 Luna (Max) | 30.5% | $0.0768 | 433.9 | 2026-11-14; AA Index v5.0.1 |
| MiMo-V2.6-Flash | 29.5% | $0.0616 | 2992.0 | 2026-11-14; AA Index v5.0.1 |
| Muse Spark Open (Xhigh) | 29.5% | $0.5069 | 457.7 | 2026-11-14; AA Index v5.0.1 |
| Qwen3.8 Max (0902) | 29.5% | $1.7571 | 2222.8 | 2026-11-14; AA Index v5.0.1 |
| DeepSeek V4 Pro 0813 (High) | 28.5% | $0.5326 | 1153.9 | 2026-11-14; AA Index v5.0.1 |
| MiniMax-M3 | 28.0% | $0.3349 | 1143.7 | 2026-11-14; AA Index v5.0.1 |
| DeepSeek V4.1 Flash 1017 (High) | 27.5% | $0.1802 | 657.8 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Luna (Medium) | 27.5% | $0.0055 | 88.0 | 2026-11-14; AA Index v5.0.1 |
| Mistral Large 4 (High)Mistral API and open weights | 27.0% | $0.3157 | 761.0 | 2026-11-14; AA Index v5.0.1 |
| Beam 1.0 | 26.5% | $0.0887 | 160.9 | 2026-11-14; AA Index v5.0.1 |
| Agnes 2.5 Pro Theta | 26.0% | $0.1357 | 2169.4 | 2026-11-14; AA Index v5.0.1 |
| Claude Sonnet 5.5 (Low, Default Fallback) | 26.0% | $0.0731 | 37.1 | 2026-11-14; AA Index v5.0.1 |
| DeepSeek V4.1 Flash 1017 (Max) | 25.0% | $0.2000 | 735.4 | 2026-11-14; AA Index v5.0.1 |
| GLM-5.3-Flash | 24.5% | $0.3556 | 8256.4 | 2026-11-14; AA Index v5.0.1 |
| Mistral Large 4 PreviewRetired public preview | 24.5% | $0.7076 | 1008.7 | 2026-11-14; AA Index v5.0.1 |
| Qwen3.8-Flash-Next | 24.5% | $0.0785 | 1306.6 | 2026-11-14; AA Index v5.0.1 |
| DeepSeek V4 Pro 0813 (Max) | 23.5% | $0.8902 | 1792.0 | 2026-11-14; AA Index v5.0.1 |
| Mistral Large 4 (Medium)Mistral API and open weights | 22.5% | $0.2011 | 548.1 | 2026-11-14; AA Index v5.0.1 |
| Claude Haiku 5.5 (Low) | 21.5% | $0.0083 | 58.7 | 2026-11-14; AA Index v5.0.1 |
| GLM-5.3 (Low) | 21.5% | $0.7577 | 1220.1 | 2026-11-14; AA Index v5.0.1 |
| JT-4.1 Flash 236B A21B | 19.0% | — | — | 2026-11-14; AA Index v5.0.1 |
| Inkling Small | 18.5% | $0.1362 | 680.8 | 2026-11-14; AA Index v5.0.1 |
| Inkling (Xhigh) | 18.0% | $0.5517 | 1090.1 | 2026-11-14; AA Index v5.0.1 |
| Solar Pro 4 | 18.0% | $0.4015 | 2440.0 | 2026-11-14; AA Index v5.0.1 |
| Apodex 1.1 | 17.0% | $0.4921 | 1028.0 | 2026-11-14; AA Index v5.0.1 |
| Qwen3.8 27B (Medium) | 16.5% | $0.7544 | 2738.2 | 2026-11-14; AA Index v5.0.1 |
| Nanbeige4.3-33B | 15.5% | — | — | 2026-11-05; AA Index v5.0.1 |
| Nemotron 3 Ultra 550B A55B (Reasoning) | 15.0% | $0.3574 | 352.8 | 2026-11-14; AA Index v5.0.1 |
| Nemotron 3.5 Super VL | 13.5% | $0.1171 | 473.9 | 2026-11-14; AA Index v5.0.1 |
| K2 Horizon 375B A23B | 12.0% | $0.0000 | 407.3 | 2026-11-14; AA Index v5.0.1 |
| GPT-5 (High) | 11.5% | — | — | 2026-11-04; AA Index v5.0.1 |
| GPT-6 Luna (Low) | 9.5% | $0.0019 | 30.6 | 2026-11-14; AA Index v5.0.1 |
| DeepSeek V4 Pro 0813 (Non-reasoning) | 8.5% | $0.4206 | 842.0 | 2026-11-14; AA Index v5.0.1 |
| K2 Horizon MoVA 36B A4B | 8.0% | — | — | 2026-11-14; AA Index v5.0.1 |
| Muse Glimmer (High) | 7.0% | $0.0695 | 469.9 | 2026-11-14; AA Index v5.0.1 |
| Kimi K3 (Low) | 5.5% | $0.0644 | 71.1 | 2026-11-14; AA Index v5.0.1 |
| Qwen3.8 27B (Low) | 5.5% | $0.4161 | 1660.6 | 2026-11-14; AA Index v5.0.1 |
| GPT-6 Luna (Non-reasoning) | 5.0% | $0.0011 | 16.5 | 2026-11-14; AA Index v5.0.1 |
| Gemma 4 31B (Reasoning) | 4.0% | — | — | 2026-11-14; AA Index v5.0.1 |
| gpt-oss-120b (High) | 4.0% | $0.0542 | 447.5 | 2026-11-14; AA Index v5.0.1 |
| K-EXAONE 2.0 | 4.0% | — | — | 2026-11-13; AA Index v5.0.1 |
| Qwen3.8 27B (Xhigh) | 4.0% | $0.5249 | 1857.8 | 2026-11-14; AA Index v5.0.1 |
| Gemini 3.5 Flash-Lite | 3.5% | $0.1828 | 107.3 | 2026-11-14; AA Index v5.0.1 |
| Gemini 3.1 Flash-Lite | 3.0% | $0.0959 | 151.2 | 2026-11-14; AA Index v5.0.1 |
| Command A+ | 2.5% | $0.0000 | 318.3 | 2026-11-14; AA Index v5.0.1 |
| Qwen3.8 27B (Non-reasoning) | 2.0% | $0.2915 | 779.8 | 2026-11-14; AA Index v5.0.1 |
| Claude 4.5 Haiku (Reasoning) | 1.5% | $0.2845 | 328.4 | 2026-11-14; AA Index v5.0.1 |
| Granite 4.2 30B | 1.5% | — | — | 2026-11-14; AA Index v5.0.1 |
| K2 Horizon 7B | 1.5% | — | — | 2026-11-14; AA Index v5.0.1 |
| Nemotron 3.5 Lightning | 1.5% | $0.0772 | 422.5 | 2026-11-14; AA Index v5.0.1 |
| DeepSeek V4.1 Flash 1017 (Non-reasoning) | 1.0% | $0.0430 | 141.4 | 2026-11-14; AA Index v5.0.1 |
| K2 Horizon 3.7B | 1.0% | — | — | 2026-11-14; AA Index v5.0.1 |
| North Mini Code | 1.0% | $0.0000 | 343.0 | 2026-11-14; AA Index v5.0.1 |
| Mistral Medium 3.5 | 0.5% | $0.7638 | 481.5 | 2026-11-14; AA Index v5.0.1 |
| Granite 4.2 3B | 0.0% | — | — | 2026-11-14; AA Index v5.0.1 |
| Granite 4.2 8B | 0.0% | $0.0072 | 249.7 | 2026-11-14; AA Index v5.0.1 |
| Mistral Small 4 (Reasoning) | 0.0% | $0.0077 | 78.3 | 2026-11-14; AA Index v5.0.1 |
| NVIDIA Nemotron 3 Nano 4B | 0.0% | — | — | 2026-11-14; AA Index v5.0.1 |
How to compare these scores
Compare rows within this version and subset. A score from the full 70-task set, an earlier CritPt version, or a different tool and grading setup cannot establish improvement or regression against this hard-subset board. The benchmark score is also distinct from the aggregate AA Intelligence Index and the Klu Index.
For broader scientific workflows that require producing files in a terminal, see Terminal-Bench Science. For academic question answering, see HLE-Diamond.