MuSR Benchmark

What is the MuSR Benchmark?

MuSR (Multistep Soft Reasoning) is a benchmark designed to evaluate the reasoning capabilities of LLMs through complex, multistep tasks specified in natural language narratives. It addresses the limitations of existing benchmarks by introducing sophisticated natural language and complex reasoning challenges.

Key features of MuSR include:

• Diverse task domains — MuSR spans various domains such as murder mysteries, object placements, and team allocation, providing a comprehensive assessment of model capabilities.
• Complex reasoning requirements — Tasks are designed to be challenging, requiring deep reasoning and problem-solving skills.
• High-quality narratives — The dataset includes well-crafted natural language narratives to ensure clarity and accuracy.
• Enhanced validation — A rigorous review process minimizes incorrect answers and improves overall dataset quality.
• Multistep reasoning tasks — Tasks require models to break down complex problems into multiple steps, testing their chain-of-thought capabilities.

MuSR serves as a robust tool to evaluate AI performance across a wide range of disciplines, from social and physical deductive reasoning to observational and constraint reasoning. It provides a comprehensive assessment of an LLM's reasoning capabilities and problem-solving skills.

Recent reasoning datasets used for benchmarking LLMs

The benchmark allows for testing and comparing various state-of-the-art language models, including but not limited to OpenAI's GPT-4, Google's Gemini, Anthropic's Claude, and Mistral AI's models.

Researchers and AI teams can utilize MuSR for in-depth evaluations when developing, fine-tuning, or benchmarking language models, especially when significant modifications are made to foundation models. This enhanced benchmark offers a more nuanced and challenging assessment of an LLM's true capabilities and limitations.

MuSR Evaluation Results

Historical raw per-task evidence retained under its original reporting format.

Updated July 24, 2024

Additional context for the historical raw results

The snapshot records raw task accuracy under the original MuSR evaluation format: Murder Mysteries (MM), Object Placements (OP), and Team Allocation (TA). The task definitions and original protocol come from the MuSR paper. The average in the second table is the arithmetic mean of those three raw task percentages.

This July 24, 2024 table is historical. It preserves the original per-task reporting format and should not be ranked against results from a different prompt template, harness version, aggregation, or normalization.

Open LLM Leaderboard fixed-harness snapshot (March 2025 data revision)

For a reproducible later comparison, the Open LLM Leaderboard evaluated MuSR through its fixed evaluation harness. The following top five entries with a recorded submission date are sorted by the normalized MUSR field in the leaderboard's contents data pinned to revision 9c09a7c. Rows without a submission date are excluded, including an undated entry tied at 38.69:

The dataset revision was published March 20, 2025, and the table records each row's submission date. It is an archived, reproducible open-model comparison under one harness rather than a universal ranking across every MuSR evaluation.

The historical MM, OP, and TA columns are raw accuracies for individual tasks, while the fixed-harness table applies a separate aggregation and chance normalization. Its raw and normalized aggregates are also distinct metrics. None of these columns can be merged into one ranking or treated as interchangeable scores.

Key Differences from Traditional Benchmarks

MuSR introduces several key enhancements over traditional benchmarks, aimed at providing a more comprehensive and challenging evaluation of language models. These improvements address limitations in existing benchmarks and create a more robust testing environment.

Partial reasoning trees

The main differences include:

• Increased answer options — MuSR expands the number of answer choices, making the evaluation more realistic and challenging. This change significantly reduces the probability of correct answers by random guessing.
• Enhanced reasoning requirements — MuSR incorporates more problems that demand complex reasoning, resulting in Chain-of-Thought (CoT) approaches outperforming traditional methods.
• Improved robustness — By increasing the number of distractors, MuSR reduces the impact of chance on correct answers, leading to greater benchmark stability.

MuSR Dataset Composition

The MuSR dataset is created through a novel neurosymbolic synthetic-to-natural generation algorithm. This process involves several key steps, beginning with Tree Template Construction. In this phase, the authors create a high-level fact set and question-answer pairs for each domain. These serve as the foundation for the reasoning trees that will be developed.

Dataset construction process for MuSR

Next, the Reasoning Tree Completion stage expands upon the initial fact set. Using recursive sampling from an LLM (in this case, GPT-4), the authors generate a tree of intermediate reasoning steps. This process creates a set of scenario-specific facts and commonsense knowledge that logically lead to the root facts.

The final stage is Story Generation, where the generated facts are embedded into a natural narrative. This is achieved through a process the authors call "chaptering," which involves generating portions of the narrative based on subsets of facts and then combining them into a coherent whole. This approach allows for the creation of longer, more complex narratives that can theoretically scale beyond the 1000-word examples in the current dataset.

CoT vs Direct Evaluation

Unlike traditional benchmarks, which favor simpler evaluation methods, MuSR requires Chain-of-Thought (CoT) reasoning to achieve better results. The following tables demonstrate the performance difference between CoT and direct (non-CoT) prompting for various models across different domains:

The impact of Chain-of-Thought (CoT) prompting varies significantly across different categories in the MuSR dataset. Some categories show substantial improvement with CoT, while others show minimal or even negative impacts. These variations highlight the diverse nature of reasoning required across different disciplines and underscore the importance of tailored approaches in language model prompting.

As evident from the data, the performance drops significantly without chain-of-thought reasoning. This substantial difference underscores the challenging nature of the MuSR dataset and highlights the importance of CoT reasoning in achieving optimal results across various disciplines.

How do the results compare: Traditional Benchmarks vs. MuSR

The table below compares the performance of various language models on traditional benchmarks and the more challenging MuSR dataset:

We can observe significant variations in performance drops across different models when tested on MuSR compared to traditional benchmarks:

• GPT-4 shows the smallest decrease, with only a 16.15% drop in performance.
• Other models experience more substantial declines, ranging from 18.35% to 26.39%.
• Some models reportedly experience even larger drops, exceeding 30%.

These results highlight the challenging nature of the MuSR dataset and the need for continued research into more sophisticated reasoning techniques for LLMs.

Additional protocol interpretation

Across published results, models generally score lower on MuSR than on traditional benchmarks such as MMLU because MuSR emphasizes multistep narrative reasoning. The historical direct-versus-CoT results above show that prompting protocol can materially change MuSR performance, while the fixed-harness snapshot provides a reproducible comparison under one later setup. Numerical gaps across MMLU, original MuSR reporting, and the Open LLM Leaderboard remain protocol-specific and should not be interpreted as one shared capability scale.