CritPt v2 Hard

Research-level physics problems from CritPt v2’s 40-task hard subset, evaluated with code execution and the official grader

SciencePass@1

Stephen M. Walker II · Co-Founder / CEO · November 14, 2026 · 1 min read

What does CritPt v2 Hard measure?

CritPt tests research-level physics problem solving. Artificial Analysis evaluates the 40-task hard subset of the 70-task CritPt v2 benchmark. Models can use Python and code execution, and the official grader checks their answers. The published metric is pass@1 over five repeats. These settings come from Artificial Analysis's intelligence benchmarking methodology.

Current results

This current snapshot keeps every published model configuration separate. Missing scores are omitted. Cost and time columns, when reported, belong to this benchmark in the same model observation; they are not costs for the complete Intelligence Index.

Model configurationScoreCost per task (USD)Time per task (seconds)Observation and source
Claude Fable 5.2 (Max, Default Fallback)76.5%$6.35301225.62026-11-14; AA Index v5.0.1
Claude Fable 5.2 (Xhigh, Default Fallback)73.5%$2.4721500.92026-11-14; AA Index v5.0.1
Claude Fable 5.2 (High, Default Fallback)73.0%$1.0494206.22026-11-14; AA Index v5.0.1
GPT-6 Astra (Medium)72.0%$0.4067161.02026-11-14; AA Index v5.0.1
GPT-6 Astra (High)71.5%$0.7142276.82026-11-14; AA Index v5.0.1
GPT-6 Astra (Xhigh)71.0%$0.8493367.62026-11-14; AA Index v5.0.1
GPT-6 Astra (Max)70.5%$1.1377509.82026-11-14; AA Index v5.0.1
GPT-6.1 Sol (Max)69.5%$0.2300391.92026-11-14; AA Index v5.0.1
GPT-6.1 Sol (Xhigh)69.5%$0.1671288.02026-11-14; AA Index v5.0.1
Gemini 4 Carbon (High)69.0%$2.43801992.12026-11-14; AA Index v5.0.1
Gemini 4 Carbon (Medium)69.0%$1.92991668.42026-11-14; AA Index v5.0.1
GPT-6.1 Sol (High)68.5%$0.1345225.12026-11-14; AA Index v5.0.1
Claude Opus 5.5 (Max, Default Fallback)68.0%$2.5767854.92026-11-14; AA Index v5.0.1
GPT-6.1 Terra (Max)68.0%$0.2505564.72026-11-14; AA Index v5.0.1
Claude Opus 5.5 (Xhigh, Default Fallback)67.5%$0.9657332.02026-11-14; AA Index v5.0.1
GPT-6.1 Sol (Medium)67.0%$0.0714112.62026-11-14; AA Index v5.0.1
GPT-6 Sol (Max)66.5%$0.3840249.32026-11-14; AA Index v5.0.1
Gemini 4 Hydrogen (Xhigh)66.0%$1.8995980.62026-11-14; AA Index v5.0.1
Gemini 4 Carbon (Low)65.0%$1.91231951.92026-11-14; AA Index v5.0.1
GPT-6 Astra (Low)63.0%$0.236790.72026-11-14; AA Index v5.0.1
GPT-6.1 Sol (Low)62.5%$0.043368.42026-11-14; AA Index v5.0.1
Claude Fable 5.1 (High, Default Fallback)62.0%$2.3876445.32026-11-14; AA Index v5.0.1
Claude Fable 5.2 (Medium, Default Fallback)61.5%$0.366975.42026-11-14; AA Index v5.0.1
Gemini 4 Hydrogen (High)61.5%$0.8777480.92026-11-14; AA Index v5.0.1
GPT-6.1 Terra (High)61.5%$0.0691166.12026-11-14; AA Index v5.0.1
Gemini 4 Hydrogen (Medium)61.0%$0.3744211.92026-11-14; AA Index v5.0.1
GPT-6.1 Terra (Xhigh)60.5%$0.1177276.02026-11-14; AA Index v5.0.1
Claude Fable 5.1 (Xhigh, Default Fallback)60.0%$4.3322767.02026-11-14; AA Index v5.0.1
Claude Fable 5.2 (Low, Default Fallback)60.0%$0.313058.52026-11-14; AA Index v5.0.1
GPT-5.6 Sol (Max)60.0%——2026-11-14; AA Index v5.0.1
Claude Opus 5.5 (High, Default Fallback)59.5%$0.3844135.82026-11-14; AA Index v5.0.1
Claude Sonnet 5.5 (Max, Default Fallback)59.0%$1.92671024.22026-11-14; AA Index v5.0.1
Claude Fable 5.1 (Medium, Default Fallback)58.5%$1.3829257.52026-11-14; AA Index v5.0.1
Claude Sonnet 5.5 (Xhigh, Default Fallback)57.5%$0.4478244.42026-11-14; AA Index v5.0.1
GPT-6.1 Terra (Medium)56.5%$0.040598.52026-11-14; AA Index v5.0.1
Muse Wonder Full (Max)Muse Code subscription56.5%——2026-11-05; AA Index v5.0.1
Claude Opus 5.5 (Medium, Default Fallback)54.0%$0.251289.72026-11-14; AA Index v5.0.1
Gemini 4 Argon (Medium)53.5%$0.7824—2026-11-14; AA Index v5.0.1
Claude Fable 5.1 (Max, Default Fallback)52.5%$4.5529808.92026-11-14; AA Index v5.0.1
Gemini 4 Argon (High)52.5%$1.17441100.92026-11-14; AA Index v5.0.1
Claude Sonnet 5.5 (High, Default Fallback)51.5%$0.180594.62026-11-14; AA Index v5.0.1
Muse Wonder (High)Meta Model API and Muse Code51.0%$1.1682422.72026-11-14; AA Index v5.0.1
GPT-5.6 Terra (Max)50.5%$0.6236385.22026-11-14; AA Index v5.0.1
GPT-6.1 Terra (Low)50.5%$0.017840.42026-11-14; AA Index v5.0.1
Muse Wonder (Max)Meta Model API and Muse Code50.5%$1.8136604.02026-11-14; AA Index v5.0.1
Claude Opus 5 (Max)50.0%$2.9047996.52026-11-14; AA Index v5.0.1
Claude Fable 5.1 (Low, Default Fallback)49.5%$0.8923162.12026-11-14; AA Index v5.0.1
Kimi K3.1 Pro49.5%$0.70171582.52026-11-14; AA Index v5.0.1
Gemini 4 Hydrogen (Low)48.0%$0.122571.42026-11-14; AA Index v5.0.1
Muse Spark 1.3 (Max)48.0%$0.6570676.02026-11-14; AA Index v5.0.1
Claude Haiku 5.5 (Xhigh)47.5%$0.0594372.92026-11-14; AA Index v5.0.1
Gemini 3.8 Flash (High)46.5%$1.1535588.82026-11-14; AA Index v5.0.1
Claude Haiku 5.5 (High)45.0%$0.0334213.22026-11-14; AA Index v5.0.1
Gemini 3.8 Flash (Medium)45.0%$0.8267458.62026-11-14; AA Index v5.0.1
Claude Haiku 5.5 (Max)43.5%$0.1013586.52026-11-14; AA Index v5.0.1
Grok 4.7 (Low)43.5%$0.7943682.52026-11-14; AA Index v5.0.1
GPT-6 Luna (Max)42.5%$0.0325464.82026-11-14; AA Index v5.0.1
Kimi K3 (Max)40.5%——2026-11-14; AA Index v5.0.1
Grok 4.6 (Xhigh)40.0%$0.8930879.02026-11-14; AA Index v5.0.1
Muse Ember (Max)39.0%$0.0538184.02026-11-14; AA Index v5.0.1
Tencent Hy439.0%$0.3076867.52026-11-14; AA Index v5.0.1
Ling 3.1 Flash 101938.5%$0.3203—2026-11-14; AA Index v5.0.1
Muse Ember (Xhigh)38.5%$0.0405139.32026-11-14; AA Index v5.0.1
GPT-6 Luna (Xhigh)37.0%$0.0129199.52026-11-14; AA Index v5.0.1
Grok 4.7 (High)37.0%$1.89411363.52026-11-14; AA Index v5.0.1
Claude Opus 5.5 (Low, Default Fallback)36.0%$0.119141.82026-11-14; AA Index v5.0.1
Inkling 2 Small35.0%$0.24871783.72026-11-14; AA Index v5.0.1
MiMo-V2.6-Pro34.0%$0.16522074.12026-11-14; AA Index v5.0.1
Claude Sonnet 5.5 (Medium, Default Fallback)33.5%$0.092651.72026-11-14; AA Index v5.0.1
Gemini 3.8 Flash (Low)33.5%——2026-11-14; AA Index v5.0.1
GLM-5.3 (Max)33.5%$1.54262133.02026-11-14; AA Index v5.0.1
GPT-6.1 Terra (Non-reasoning)33.5%$0.017032.22026-11-14; AA Index v5.0.1
Grok 4.7 (Xhigh)33.5%$2.23691656.72026-11-14; AA Index v5.0.1
MiniMax-M3.1-Flash32.5%$0.16651185.22026-11-14; AA Index v5.0.1
Step 5 Preview32.5%$0.3584527.42026-11-14; AA Index v5.0.1
GPT-6 Luna (High)31.0%$0.0096148.62026-11-14; AA Index v5.0.1
Claude Haiku 5.5 (Medium)30.5%$0.0167111.42026-11-14; AA Index v5.0.1
GPT-5.6 Luna (Max)30.5%$0.0768433.92026-11-14; AA Index v5.0.1
MiMo-V2.6-Flash29.5%$0.06162992.02026-11-14; AA Index v5.0.1
Muse Spark Open (Xhigh)29.5%$0.5069457.72026-11-14; AA Index v5.0.1
Qwen3.8 Max (0902)29.5%$1.75712222.82026-11-14; AA Index v5.0.1
DeepSeek V4 Pro 0813 (High)28.5%$0.53261153.92026-11-14; AA Index v5.0.1
MiniMax-M328.0%$0.33491143.72026-11-14; AA Index v5.0.1
DeepSeek V4.1 Flash 1017 (High)27.5%$0.1802657.82026-11-14; AA Index v5.0.1
GPT-6 Luna (Medium)27.5%$0.005588.02026-11-14; AA Index v5.0.1
Mistral Large 4 (High)Mistral API and open weights27.0%$0.3157761.02026-11-14; AA Index v5.0.1
Beam 1.026.5%$0.0887160.92026-11-14; AA Index v5.0.1
Agnes 2.5 Pro Theta26.0%$0.13572169.42026-11-14; AA Index v5.0.1
Claude Sonnet 5.5 (Low, Default Fallback)26.0%$0.073137.12026-11-14; AA Index v5.0.1
DeepSeek V4.1 Flash 1017 (Max)25.0%$0.2000735.42026-11-14; AA Index v5.0.1
GLM-5.3-Flash24.5%$0.35568256.42026-11-14; AA Index v5.0.1
Mistral Large 4 PreviewRetired public preview24.5%$0.70761008.72026-11-14; AA Index v5.0.1
Qwen3.8-Flash-Next24.5%$0.07851306.62026-11-14; AA Index v5.0.1
DeepSeek V4 Pro 0813 (Max)23.5%$0.89021792.02026-11-14; AA Index v5.0.1
Mistral Large 4 (Medium)Mistral API and open weights22.5%$0.2011548.12026-11-14; AA Index v5.0.1
Claude Haiku 5.5 (Low)21.5%$0.008358.72026-11-14; AA Index v5.0.1
GLM-5.3 (Low)21.5%$0.75771220.12026-11-14; AA Index v5.0.1
JT-4.1 Flash 236B A21B19.0%——2026-11-14; AA Index v5.0.1
Inkling Small18.5%$0.1362680.82026-11-14; AA Index v5.0.1
Inkling (Xhigh)18.0%$0.55171090.12026-11-14; AA Index v5.0.1
Solar Pro 418.0%$0.40152440.02026-11-14; AA Index v5.0.1
Apodex 1.117.0%$0.49211028.02026-11-14; AA Index v5.0.1
Qwen3.8 27B (Medium)16.5%$0.75442738.22026-11-14; AA Index v5.0.1
Nanbeige4.3-33B15.5%——2026-11-05; AA Index v5.0.1
Nemotron 3 Ultra 550B A55B (Reasoning)15.0%$0.3574352.82026-11-14; AA Index v5.0.1
Nemotron 3.5 Super VL13.5%$0.1171473.92026-11-14; AA Index v5.0.1
K2 Horizon 375B A23B12.0%$0.0000407.32026-11-14; AA Index v5.0.1
GPT-5 (High)11.5%——2026-11-04; AA Index v5.0.1
GPT-6 Luna (Low)9.5%$0.001930.62026-11-14; AA Index v5.0.1
DeepSeek V4 Pro 0813 (Non-reasoning)8.5%$0.4206842.02026-11-14; AA Index v5.0.1
K2 Horizon MoVA 36B A4B8.0%——2026-11-14; AA Index v5.0.1
Muse Glimmer (High)7.0%$0.0695469.92026-11-14; AA Index v5.0.1
Kimi K3 (Low)5.5%$0.064471.12026-11-14; AA Index v5.0.1
Qwen3.8 27B (Low)5.5%$0.41611660.62026-11-14; AA Index v5.0.1
GPT-6 Luna (Non-reasoning)5.0%$0.001116.52026-11-14; AA Index v5.0.1
Gemma 4 31B (Reasoning)4.0%——2026-11-14; AA Index v5.0.1
gpt-oss-120b (High)4.0%$0.0542447.52026-11-14; AA Index v5.0.1
K-EXAONE 2.04.0%——2026-11-13; AA Index v5.0.1
Qwen3.8 27B (Xhigh)4.0%$0.52491857.82026-11-14; AA Index v5.0.1
Gemini 3.5 Flash-Lite3.5%$0.1828107.32026-11-14; AA Index v5.0.1
Gemini 3.1 Flash-Lite3.0%$0.0959151.22026-11-14; AA Index v5.0.1
Command A+2.5%$0.0000318.32026-11-14; AA Index v5.0.1
Qwen3.8 27B (Non-reasoning)2.0%$0.2915779.82026-11-14; AA Index v5.0.1
Claude 4.5 Haiku (Reasoning)1.5%$0.2845328.42026-11-14; AA Index v5.0.1
Granite 4.2 30B1.5%——2026-11-14; AA Index v5.0.1
K2 Horizon 7B1.5%——2026-11-14; AA Index v5.0.1
Nemotron 3.5 Lightning1.5%$0.0772422.52026-11-14; AA Index v5.0.1
DeepSeek V4.1 Flash 1017 (Non-reasoning)1.0%$0.0430141.42026-11-14; AA Index v5.0.1
K2 Horizon 3.7B1.0%——2026-11-14; AA Index v5.0.1
North Mini Code1.0%$0.0000343.02026-11-14; AA Index v5.0.1
Mistral Medium 3.50.5%$0.7638481.52026-11-14; AA Index v5.0.1
Granite 4.2 3B0.0%——2026-11-14; AA Index v5.0.1
Granite 4.2 8B0.0%$0.0072249.72026-11-14; AA Index v5.0.1
Mistral Small 4 (Reasoning)0.0%$0.007778.32026-11-14; AA Index v5.0.1
NVIDIA Nemotron 3 Nano 4B0.0%——2026-11-14; AA Index v5.0.1
Artificial AnalysisPython/code execution, official grader, 5 repeatsCritPt v2 Hard, 40 tasks from the 70-task full setPass@1artificialanalysis.ai

How to compare these scores

Compare rows within this version and subset. A score from the full 70-task set, an earlier CritPt version, or a different tool and grading setup cannot establish improvement or regression against this hard-subset board. The benchmark score is also distinct from the aggregate AA Intelligence Index and the Klu Index.

For broader scientific workflows that require producing files in a terminal, see Terminal-Bench Science. For academic question answering, see HLE-Diamond.