Reasoning Models Explained: Chain-of-Thought, Test-Time Compute, and Thinking Tokens

Ask a standard chat model a hard logic puzzle and it will often blurt out a confident, wrong answer in the first few tokens. Ask a reasoning model the same question and it will spend anywhere from a few hundred to tens of thousands of tokens working through the problem before committing to a final answer. Same underlying architecture — a decoder-only transformer — but a completely different relationship between compute and quality.

This shift, often summarized as “test-time compute,” is one of the more consequential changes in how LLMs are trained and served. It is easy to confuse with chain-of-thought prompting, which has existed for years. The two are related but not the same thing, and the difference matters if you are deciding which model to put behind a product.

From Prompting Trick to Trained Behavior

Chain-of-thought (CoT) prompting started as a discovery about in-context behavior: if you ask a model to “think step by step” before answering, accuracy on multi-step problems improves noticeably. Nothing about the model changes — you are just conditioning the same weights to produce intermediate tokens before the final answer, and those intermediate tokens act as extra computation the model can attend back to.

The limitation is that a model never explicitly trained to reason well will often produce reasoning that looks plausible but doesn’t actually constrain the final answer. It writes steps, then answers as if it hadn’t written them.

Reasoning models close that gap by training directly on the process, not just the outcome. The recipe generally looks like this:

1. Generate many candidate reasoning traces for a given problem.
2. Score the traces (or just the final answers) against a verifier — unit tests for code, a symbolic checker for math, an exact-match grader for factual questions.
3. Use reinforcement learning to reward traces that lead to correct, verifiable answers, and penalize traces that don’t.
4. Repeat, so the model learns to allocate more tokens to harder problems and fewer to easy ones.

The result is a model that has learned, from experience, that pausing to explore a wrong branch and backtrack is often worth the extra tokens. That is qualitatively different from a base model that was merely told to “think step by step” in the prompt.

What “Thinking Tokens” Actually Are

Under the hood, a reasoning model’s output is still just a token sequence. What changes is that a chunk of that sequence — often wrapped in a delimiter like a <thinking> block or simply generated in a separate channel — is treated as scratch space rather than the final answer.

Two things typically happen with this scratch space in production APIs:

• It is billed as output tokens even though the user never asked for it, because it still costs a forward pass per token.
• It is hidden or summarized in the UI, because raw reasoning traces are often repetitive, occasionally reveal safety-relevant deliberation, or are just not pleasant to read.

A minimal way to think about the token budget for a single response:

total_output_tokens = thinking_tokens + answer_tokens
total_cost ≈ total_output_tokens × price_per_output_token
total_latency ≈ total_output_tokens / tokens_per_second

This is why reasoning models feel slower and pricier per response even when the final answer is short — you are paying for a lot of tokens the user never sees.

The Test-Time Compute Curve

The core empirical finding behind reasoning models is that, for a fixed trained model, accuracy on hard tasks keeps improving as you let it generate more reasoning tokens — up to a point, after which returns diminish and eventually flatten. This gives providers a knob that didn’t really exist before: instead of only scaling quality by training a bigger model, you can scale it at inference time by letting a smaller model “think longer.”

This is usually exposed as a reasoning effort or thinking budget parameter:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-reasoning-model",
    max_tokens=4096,
    thinking={"type": "enabled", "budget_tokens": 2048},
    messages=[
        {"role": "user", "content": "A train leaves station A at 60 km/h, another leaves station B 300km away at 90 km/h heading toward it. When do they meet?"}
    ],
)

The exact parameter names differ across providers (reasoning_effort, thinking_budget, max_thinking_tokens), but the concept is consistent: you are trading latency and cost for a shot at higher accuracy on problems that benefit from deliberation.

Crucially, this curve is task-dependent. Simple factual lookups or short rewrites don’t benefit from more thinking tokens — the model reaches the correct answer immediately and additional tokens just restate the same conclusion, or worse, start second-guessing a correct answer into a wrong one. This is sometimes called “overthinking,” and it’s a real failure mode: giving a reasoning model an unnecessarily large budget on an easy question can reduce accuracy.

Budget Forcing and Controlling How Much a Model Thinks

Because unconstrained reasoning models will sometimes generate an enormous number of thinking tokens on a problem that doesn’t warrant it, most serving stacks implement some form of budget control:

• Hard token caps — truncate the thinking block after N tokens and force an answer.
• Budget forcing — append a token like “Final answer:” once the budget is exhausted, nudging the model to wrap up even mid-thought.
• Effort tiers — expose a small number of presets (low / medium / high) rather than a raw token count, which is easier for application developers to reason about and easier for the provider to route efficiently across hardware.

A practical pattern for a production endpoint that serves mixed traffic:

def pick_reasoning_effort(request):
    if request.task_type in ("classification", "extraction", "short_qa"):
        return "low"
    if request.task_type in ("multi_step_math", "code_debugging", "planning"):
        return "high"
    return "medium"

Routing effort by task type, rather than giving every request the same budget, is one of the highest-leverage things you can do to control reasoning-model spend.

Self-Consistency: Reasoning Without a Reasoning Model

Before trained reasoning models existed, a cheaper trick called self-consistency approximated some of the same benefit: sample the same prompt N times with non-zero temperature, extract the final answer from each sample, and take a majority vote.

from collections import Counter

def self_consistency_answer(client, prompt, n=5):
    answers = []
    for _ in range(n):
        resp = client.messages.create(
            model="claude-base-model",
            max_tokens=1024,
            temperature=0.7,
            messages=[{"role": "user", "content": prompt + "\nThink step by step, then give a final answer."}],
        )
        answers.append(extract_final_answer(resp))
    return Counter(answers).most_common(1)[0][0]

This still works today and is worth knowing even if you use a reasoning model, because it composes: sampling multiple reasoning traces from a reasoning model and voting across them can push accuracy higher still, at N times the cost.

Process Reward Models, Briefly

A related but distinct idea is training a process reward model (PRM) — a separate model that scores each individual step of a reasoning trace, not just the final answer. PRMs are used to guide search over multiple candidate reasoning paths (for example, tree-of-thought style expansion, where you generate several next steps and use the PRM to prune the weaker ones before continuing). This is more compute-intensive to serve than a single reasoning model doing one pass, and mostly shows up in research pipelines or specialized high-value tasks like formal math and competitive programming, rather than general-purpose chat products.

Prompted CoT vs a Trained Reasoning Model

Aspect	Prompted chain-of-thought	Trained reasoning model
How it’s produced	Prompt instructs the base model to “think step by step”	Model is trained via RL against verifiable rewards to reason
Reliability of the reasoning-to-answer link	Weak — reasoning can be decorative	Stronger — training rewards traces that actually lead to correct answers
Compute control	None built in; you control length via prompt only	Explicit budget/effort parameter
Cost model	Normal output tokens	Normal output tokens plus (often billed) thinking tokens
Works on any model	Yes, any instruction-following model	No, requires a model specifically trained for it
Best for	Cheap, general tasks with moderate reasoning needs	Hard multi-step math, code debugging, planning, agentic tool-use chains

When Reasoning Models Are Worth the Latency

In practice, the decision comes down to whether a task has a verifiable structure — a right answer that can be checked, even approximately. Reasoning models shine on math, code, structured planning, and multi-step agentic workflows where an intermediate mistake compounds. They add little value, and meaningful latency, on tasks like casual conversation, summarization of a short document, or simple classification, where a well-tuned base model with a good prompt gets you most of the way there for a fraction of the cost.

A reasonable default architecture for a product serving mixed workloads is to route: use a fast, non-reasoning model as the default, detect when a request looks like it needs multi-step reasoning (math, code, “plan out,” “debug why”), and only then escalate to a reasoning model with an appropriate budget. This keeps median latency low while still catching the requests that actually benefit from extra thinking.

Key Takeaways

• Chain-of-thought prompting nudges a base model to produce intermediate steps; a trained reasoning model is optimized end-to-end, via RL against verifiable rewards, to make those steps actually load-bearing.
• “Thinking tokens” are ordinary output tokens generated in a scratch space; they are usually billed and often hidden or summarized in the UI.
• Test-time compute lets you trade latency and cost for accuracy on a per-request basis, but the returns flatten and can even reverse on easy tasks (“overthinking”).
• Budget forcing and effort tiers (low/medium/high) are the practical knobs for controlling how much a model thinks, and routing by task type is the highest-leverage lever you have.
• Self-consistency (sample-and-vote) is a cheaper, model-agnostic approximation of some of the same benefit, and it composes with trained reasoning models.
• Process reward models score individual reasoning steps rather than final answers, enabling search-based methods, but are mostly a research and high-value-task tool today.
• The right default is to route: fast model for most traffic, reasoning model with a scoped budget only for requests that show verifiable, multi-step structure.