Test-Time Compute (Inference Scaling)

What is Test-Time Compute?

Test-time compute, also called inference scaling, is the practice of giving a model additional computation, more time and more tokens, during generation so it can work through harder problems. Instead of producing an answer immediately, a model using test-time compute deliberates, plans, and checks its own logic before returning a final response.

Historically, large language model capability came almost entirely from pretraining and fine-tuning. Compute spent at inference was fixed, bounded mostly by the length of the prompt and the length of the output. Test-time compute changes that. By letting a model generate thousands of internal reasoning tokens, often hidden from the end user, before producing its final output, it can substantially improve performance on tasks that require careful logic, math, or complex coding.

This resembles the difference between human System 1 and System 2 thinking: System 1 is fast and intuitive, System 2 is slow and deliberate. Test-time compute gives a model something like a System 2 mode, trading latency and cost for accuracy and depth.

The lineage runs from OpenAI's o1 (the first widely deployed reasoning model, late 2024) and DeepSeek-R1 through GPT-5 to today's models, where reasoning is a built-in, budgeted capability: the GPT-5.6 family's reasoning effort levels, Claude's adaptive thinking, Gemini 3's thinking levels, and DeepSeek V4's thinking mode.

Pretraining Scaling vs. Inference-Time Scaling Laws

For years the AI industry worked from pretraining scaling laws: performance increases as a power-law function of model size, dataset size, and training compute. Those laws still hold, but training ever larger models runs into physical, financial, and data-scarcity limits.

Inference-time scaling laws offer another path to better performance, and in some cases a cheaper one. Research has shown that giving a smaller model a much larger inference compute budget can let it outperform larger models running standard inference.

The core idea is that performance can reliably improve as a function of compute spent at test time. If a model generates a solution, critiques it, searches alternative paths, and iterates, the odds of reaching the correct answer go up. This means organizations can invest compute at runtime instead of spending hundreds of millions of dollars solely on pretraining larger models. It allows for flexible scaling: minimal compute for trivial tasks, heavy compute for problems that need to get the answer right.

Key Mechanics: Chain-of-Thought Search, Process Reward Models (PRMs), and Monte Carlo Tree Search

Test-time compute relies on a few underlying mechanisms that guide a model's deliberation.

Chain-of-Thought Search

Chain-of-thought (CoT) prompting first showed that asking a model to explain its reasoning step by step improves accuracy. Test-time compute extends this idea. Instead of a single, linear chain of thought, a model explores many possible intermediate reasoning steps, generating multiple candidate "next steps" and building a branching tree of thoughts.

Process Reward Models (PRMs)

To navigate that tree, systems often use process reward models (PRMs). Unlike outcome reward models (ORMs), which score only the final answer, PRMs score the correctness of individual reasoning steps. As a model "thinks," the PRM scores intermediate steps, penalizing faulty logic or hallucination early and rewarding sound deductive steps. This step-by-step check keeps errors from compounding over long reasoning chains.

Monte Carlo Tree Search (MCTS)

Combining chain-of-thought search with PRMs lets test-time compute use search algorithms such as Monte Carlo Tree Search (MCTS). MCTS comes from reinforcement learning and was used in AlphaGo. It balances exploration, trying new and unproven reasoning paths, with exploitation, extending paths the PRM scores highly. The model searches through candidate solutions, backtracking when a path looks wrong, until it settles on an answer.

Latency, Token Multipliers, and Cost Trade-offs

Spending more compute at test time changes the unit economics and user experience of an AI application. Reasoning models use far more tokens than instruction models because they generate internal reasoning tokens before producing visible output.

This creates a trade-off between reasoning depth, latency, and cost. The table below gives illustrative orders of magnitude, not measured figures. Actual numbers vary by provider, model, and prompt.

Engineers need to manage these multipliers carefully. A "Medium" reasoning budget might cost 20 to 50 times more per request than standard generation, because the model is generating thousands of hidden tokens. The latency, often tens of seconds or minutes, calls for different UI/UX approaches, such as asynchronous processing or progressive loading indicators, so users aren't left staring at a blank screen.

Provider Budget Controls (August 2026)

Every major API now exposes a knob for test-time compute, but the mechanisms differ, and the industry has been converging from raw token budgets toward discrete effort levels where the model decides how much to think.

• OpenAI's reasoning guide (developers.openai.com, checked August 2026) documents a reasoning.effort parameter whose supported values are model-dependent and drawn from none, minimal, low, medium, high, xhigh, and max; GPT-5.5 and the GPT-5.6 family default to medium. Reasoning tokens are billed as output tokens and reported in the response under usage.output_tokens_details.reasoning_tokens.
• Anthropic's adaptive thinking docs describe a mid-migration between two schemes. Models that support only extended thinking (Claude Sonnet 4.5, Opus 4.5, Haiku 4.5, and earlier Claude 4 models) use manual thinking: {type: "enabled", budget_tokens: N}, with a 1,024-token minimum, the budget counting toward max_tokens, and the budget acting as a target rather than a strict cap. Claude 4.7 and later models reject that configuration entirely (a 400 error) and use adaptive thinking (thinking: {type: "adaptive"}) steered by output_config: {effort: ...} (default high); at lower effort the model may skip thinking altogether on easy inputs. Thinking tokens bill as output and appear in usage.output_tokens_details.thinking_tokens on both schemes.
• Google's Gemini 3 developer guide used a numeric thinking_budget token count in the Gemini 2.5 series (an upper limit, defaulting to a maximum of 8,192 tokens if unset) and moved to a discrete thinking_level parameter (minimal/low/medium/high, defaulting to high) with Gemini 3, matching the industry shift from token budgets to effort levels. The two parameters cannot be combined in one request.
• DeepSeek's V4 API combines a thinking: {type: "enabled"/"disabled"} toggle with a graduated reasoning_effort of low, high, or max (per DeepSeek's thinking-mode docs); requests for medium or xhigh are accepted for cross-provider compatibility and mapped to high/max. Thinking is on by default at high effort.

One operational trap worth knowing: on Anthropic, changing budget_tokens between requests on the manual scheme, or changing effort between requests on the adaptive scheme, invalidates prompt-cache breakpoints, because the resolved value is rendered into the prompt. Pick a configuration and hold it stable for the life of a cached conversation, or the caching savings quietly disappear.

Parallel vs. Sequential Scaling

Test-time compute comes in two shapes that can be combined:

• Sequential scaling extends a single reasoning trace: the model thinks longer before answering. This is what reasoning effort and thinking budgets control, and it helps most on problems where later steps depend on earlier ones, such as multi-step math and debugging.
• Parallel scaling samples multiple independent answers and selects among them, via majority vote (self-consistency), best-of-n with a reward model or verifier, or unit tests for code. Parallel scaling adds cost but not latency (samples run concurrently), which makes it attractive when wall-clock time matters more than spend. Frontier lab results on competition math and programming benchmarks have repeatedly relied on heavy parallel sampling on top of long reasoning traces.

A practical corollary: if a task has a cheap automatic verifier (tests pass, JSON validates, answer checks out numerically), parallel sampling with verification often beats simply raising the reasoning effort.

When to Use Reasoning Models vs. Standard Instruction Models

Deciding when to use test-time compute means weighing the task's requirements. Here is a decision framework for architectural choices.

Use Standard Instruction Models When:

• Latency is critical. Real-time conversational agents, autocomplete features, or synchronous API endpoints need sub-second response times.
• Tasks are deterministic or trivial. Summarization, tone adjustment, classification, and straightforward data extraction do not benefit much from deep reasoning.
• Cost sensitivity is high. High-volume, low-margin operations cannot absorb the token multipliers of reasoning models.
• Context is fully provided. If the prompt already has all necessary information and just needs formatting or extraction, standard inference is enough.

Use Reasoning (Test-Time Compute) Models When:

• Accuracy outweighs speed. Medical diagnostics, legal analysis, or financial modeling, where a hallucination or logic error has real consequences.
• The task involves planning or search. Agentic workflows that break down a large goal, execute steps, evaluate outcomes, and adjust course.
• Complex coding and mathematics. Algorithmic optimization, writing large interconnected software modules, or solving novel mathematical problems that need step-by-step verification.
• Zero-shot complex problem solving. When fine-tuning a model for a narrow, complex use case is impractical, a reasoning model with a solid prompt is often more effective.

Monitoring and Evaluating Test-Time Reasoning Traces in Production

Running reasoning models in production requires real observability. Because a model's actual work happens in the hidden reasoning trace, watching only the final output is not enough for debugging or improving the system.

Organizations need tools to log and analyze reasoning traces, typically through tracing infrastructure that captures the intermediate CoT tokens, when the provider or framework exposes them, and checks them for logical coherence. Key metrics to monitor:

1. Reasoning length vs. accuracy. Is the model actually getting more accurate with more tokens, or is it stuck in loops? Tracking the correlation between token count and task success matters here.
2. Backtracking frequency. How often does the model reverse its logic? High backtracking can mean a poorly framed prompt or a task the model is not suited for.
3. PRM score distributions. With custom routing or self-hosted PRMs, monitoring intermediate step scores can surface common logical pitfalls in a model's domain-specific reasoning.
4. Cost constraints. Budget caps and alerts need to be in place so a runaway reasoning chain does not burn through the budget.

Analyzing reasoning traces this way, alongside LLM evaluation methods, lets engineers find failure modes, refine prompts, and pick the right inference budget for each task category.

Frequently Asked Questions

Does test-time compute guarantee a correct answer? No. It significantly increases the probability of reaching a correct logical conclusion, but reasoning models can still hallucinate or follow flawed logical paths. They are also bounded by what they learned during pretraining; test-time compute cannot synthesize facts the model was never exposed to.

Can I control the amount of test-time compute a model uses? Yes. As of August 2026, OpenAI exposes reasoning.effort levels (from none up to xhigh and max depending on model), Anthropic's current models use adaptive thinking steered by an effort setting (older Claude models take an explicit budget_tokens target), and Gemini 3 uses a discrete thinking_level where the Gemini 2.5 series took a numeric thinking_budget. In every case the hidden reasoning tokens bill at the output-token rate, so budget controls are cost controls.

Does more thinking always help? No. Published evaluations and provider guidance agree that returns diminish and can invert: on easy tasks, forcing long reasoning wastes tokens and occasionally degrades accuracy through overthinking, which is why adaptive schemes that let the model skip thinking on easy inputs have become the default. Measure accuracy against reasoning length on your own task distribution before standardizing on a high effort setting.

Is test-time compute the same as agentic workflows? They are related but distinct. Test-time compute typically happens within the model's internal generation process, such as an internal chain of thought. Agentic workflows involve external loops: calling tools, executing code, retrieving documents, and running multiple models in a scaffolded loop. Reasoning models are increasingly used as the core decision engine within agentic workflows.

Do reasoning models make smaller models obsolete? On the contrary. Inference scaling laws suggest that smaller, efficient models augmented with significant test-time compute can rival or surpass massive models on complex tasks, which could make high-tier AI capability more accessible.