Test-time compute is the computing work a model does while answering a question, as distinct from the computing that went into building it. Instead of training a bigger model, a system can let the same model do more work per question: write out reasoning steps, sample several candidate answers, score them, or revise a draft. That extra work costs more time and more money per answer, so the open question is when the trade is worth making.
What is test-time compute?
Compute here just means computation: work done on hardware. Language models consume compute in two distinct places.
Training compute is the computing spent to build a model. It covers processing the training data and adjusting the model's internal parameters until the model is finished. All of that spending happens before you ever type a question.
Test-time compute, also called inference-time compute, is the computing spent when the model answers a question. The name comes from evaluation language: the moment a model is given a prompt and produces a response is the test, and any computation during that response happens at test time.
Concretely, a chat model answers by generating text one token at a time. A token is a small chunk of text, roughly a word or part of a word, that a language model reads and writes. Every token the model writes requires computation, so longer answers require more test-time compute. A model that reasons out loud, checks its work, or drafts several answers before replying spends more compute on your question than a model that replies immediately.
For years, the main way to get better answers was a bigger model or more training. Test-time compute is the other lever: let the same model do more work per question. The parameters stay fixed, and the system changes how much computation it performs at answer time.
Training compute vs test-time compute
The two kinds of compute differ in when they are spent and what they change.
Training compute is spent once and produces a fixed model. After training, the parameters do not change when you use the model. If the trained model struggles with a question, you cannot add more training compute at the moment of the question. You work with the model you have.
Test-time compute is spent every time the model answers, and the amount can vary per question. A system can answer a simple prompt briefly and a hard prompt at length. It can generate one answer or many. It can stop at a draft or check the draft and revise it. That flexibility makes test-time compute a dial rather than a fixed cost.
The two levers interact. OpenAI introduced o1 on September 12, 2024, a model trained with reinforcement learning to produce a long internal chain of thought before answering. OpenAI wrote that o1's performance "consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)." Training taught the model to reason at length, and spending compute on that reasoning at answer time improved results further.
The cost side is straightforward. More tokens generated means more time and more money per answer. A response that takes the long way around keeps hardware busy longer, and the person waiting sees a slower reply.
How do models spend more compute on an answer?
The papers below describe four broad mechanisms. Systems often combine them.
Write out intermediate reasoning steps. Chain-of-thought prompting asks a model to produce a series of intermediate reasoning steps before the final answer instead of jumping straight to a result (Wei et al.). The reasoning steps are extra tokens, and extra tokens are extra compute. This is the mechanism behind "thinking" modes that show their work.
Sample many answers and take a majority vote. Self-consistency samples a diverse set of reasoning paths rather than taking one greedy answer, then picks the final answer the paths agree on (Wang et al.). A greedy answer commits to a single path; sampling explores many paths and lets the set of drafts decide. Each sampled answer costs compute, so this multiplies the generation bill by the number of samples.
Generate many answers and score them. Another family of methods generates many candidate answers and uses a separate scoring model, called a verifier or reward model, to pick which candidate to return. This approach is called best-of-N. A related idea scores each step of a solution rather than only the final result; a scorer trained that way is called a process reward model (Lightman et al.). Compute goes into generating the candidates and into scoring them.
Revise, or keep thinking. A model can also revise its own answer, or extend its thinking past the point where it would otherwise stop. Snell et al. study adaptively updating the model's response distribution, which in practice means revising a draft rather than accepting it as is. The s1 paper uses a technique called budget forcing: end the thinking process early, or lengthen it by appending the word "Wait" when the model tries to stop, which can lead it to double-check and fix wrong steps.
All four mechanisms share a structure. The model produces more text than a direct answer would require, and then a vote, a scorer, or the model itself turns that extra text into a final answer.
For an engineering view of the same methods, including outcome and process reward models, beam search and tree search, see our technical explainer on test-time compute scaling in large language models.
What did the research find?
Chain-of-thought prompting. Wei et al., at Google, define a chain of thought as a series of intermediate reasoning steps. The paper reports that prompting a 540B-parameter model with just eight chain-of-thought examples reached state-of-the-art accuracy on GSM8K, a math word problem benchmark, at the time, surpassing finetuned GPT-3 with a verifier. The paper says these reasoning abilities emerge in sufficiently large models, meaning they appear only above a certain model size.
Self-consistency. Wang et al., at Google, replace the single greedy answer with a diverse set of sampled reasoning paths, then select the final answer that is consistent across them. The paper reports gains over chain-of-thought prompting on GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and ARC-challenge (+3.9%).
Process supervision. Lightman et al., at OpenAI, compare two ways of training the reward models used to score candidate answers. Outcome supervision gives feedback on the final result. Process supervision gives feedback on each reasoning step. The paper reports that process supervision significantly outperformed outcome supervision on the MATH dataset, and that their process-supervised model solves 78% of problems from a representative subset of the MATH test set. The team released PRM800K, 800,000 step-level human feedback labels. The finding matters for test-time compute because scoring is how many systems decide which expensive candidate answer to keep.
Where to spend a fixed budget. Snell et al. study two mechanisms: searching against dense, process-based verifier reward models, and updating the model's response distribution adaptively, that is, making revisions. The paper reports that effectiveness depends on prompt difficulty. A "compute-optimal" strategy that allocates test-time compute per prompt improved efficiency by more than 4x compared to a best-of-N baseline. The paper also reports a FLOPs-matched evaluation, which holds total computation fixed and asks where to spend it: a bigger model, or more work per answer. In that comparison, on problems where a smaller base model has somewhat non-trivial success rates, test-time compute can outperform a model 14x larger.
Reasoning trained into the model. OpenAI's o1 announcement describes a model trained with reinforcement learning to produce a long internal chain of thought before answering. On the 2024 AIME exams, OpenAI reports, o1 averaged 74% with a single sample per problem, 83% with consensus among 64 samples, and 93% when re-ranking 1000 samples with a learned scoring function. GPT-4o averaged 12%. Those three figures track the mechanisms above: one direct answer, a vote across many samples, and scoring many samples with a learned function. For the o1 series, OpenAI said it shows a model-generated summary of the chain of thought rather than the raw chain. The page now notes that o1 is deprecated in the API and o1-preview is retired.
A minimal recipe. Muennighoff et al. describe s1, which seeks a minimal approach to test-time scaling. The authors fine-tuned Qwen2.5-32B-Instruct on s1K, 1,000 questions with reasoning traces, and used budget forcing to control thinking at answer time. The paper reports that s1-32B exceeds o1-preview on competition math by up to 27% (MATH and AIME24), and that budget forcing raised AIME24 from 50% to 57%. The model, data and code are open source.
Reasoning without labeled traces. The DeepSeek-R1 paper reports that reasoning abilities can be incentivized through pure reinforcement learning, without human-labeled reasoning trajectories, with emergent self-reflection and verification. Both are forms of test-time work: the model checks its own output instead of emitting it once.
Is more thinking always better?
Not according to these sources. Three limits stand out.
Difficulty changes the trade. Snell et al. report that the effectiveness of test-time compute depends on prompt difficulty. Their result that test-time compute can outperform a model 14x larger is limited to problems where the smaller base model has somewhat non-trivial success rates. The paper does not make that claim for problems where the smaller model rarely succeeds.
Thinking costs tokens, and tokens cost time and money. Every extra reasoning step, sample and revision is more generation, which means a slower reply and a higher compute bill per answer. A system that re-ranks 1000 candidate answers spends far more than a system that answers once.
The gains do not carry to every task. In OpenAI's own human preference test, trainers preferred o1-preview over GPT-4o by a large margin in reasoning-heavy categories like data analysis, coding and math, but not on some natural language tasks. OpenAI wrote that this suggests o1-preview "is not well-suited for all use cases."
The s1 paper points the same way from the other side: budget forcing can shorten thinking as well as lengthen it. Snell et al. make a similar point with their compute-optimal strategy, which allocates compute per prompt instead of spending the same amount everywhere.
Why it matters when you use AI
If you use a chat app, you have likely seen this machinery exposed as settings. Many apps offer a "thinking" or "reasoning" mode, or an effort setting that controls how long the model works before answering. Those controls are test-time compute dials. Turning them up lets the same model spend more computation on your question.
The trade is speed and cost against accuracy on hard problems. For multi-step problems of the kind the papers study, such as math word problems and exam questions, more test-time compute can improve the answer, and the results above report the size of those gains. For simple questions, a fast mode is fine. A short factual prompt does not need sampled candidates or step-level verification, and skipping them gets you a reply sooner.
Two practical notes. A noticeably slower reply can mean the system is spending test-time compute: reasoning, sampling, scoring or revising behind the scenes. And what you see may not be the full work: in its 2024 o1 announcement, OpenAI said it shows a model-generated summary of the chain of thought rather than the raw chain. A visible "thinking" panel is not necessarily the complete reasoning.
Related explainers
- What is chain-of-thought prompting?
- What are AI scaling laws?
- What is a KV cache?
- What is speculative decoding?
