Speculative Decoding: How LLMs Generate Faster Without Losing Quality

Autoregressive generation has an awkward property: token N+1 depends on token N, so a model cannot start computing it until token N has actually been produced. Each token requires its own full forward pass through every layer of the network, and those passes cannot be parallelized across time the way training can be. This is why generation latency scales almost linearly with output length no matter how much GPU compute you throw at it — you are latency-bound by sequential dependency, not throughput-bound.

Speculative decoding attacks this by having a small, fast model guess several tokens ahead, and having the large model verify all of those guesses in a single parallel pass instead of one sequential pass per token. Remarkably, when done correctly, this produces exactly the same output distribution as running the large model alone — it is a pure speedup, not an approximation.

The Serial Bottleneck

To make the problem concrete: generating 100 tokens from a large model normally means 100 sequential forward passes, each one waiting on the previous token. Even if your GPU could theoretically process a batch of 100 tokens through the network in roughly the same wall-clock time as processing 1 token (because the bottleneck is often memory bandwidth to load weights, not raw compute), the autoregressive dependency prevents you from ever presenting 100 tokens to the model at once during normal generation — you don’t have tokens 2 through 100 yet.

Speculative decoding breaks this circular dependency by having a separate, cheap model produce a plausible guess for what tokens 2 through 100 might be, so the big model can check all of them in parallel.

The Draft-and-Verify Loop

The mechanism has two models: a small, fast draft model and the large target model whose output distribution you actually want.

1. Draft model generates k candidate tokens autoregressively (cheap, fast, sequential but small).
2. Target model runs ONE forward pass over the original context + all k candidate tokens,
   producing its own probability distribution at each of those k positions (parallel, one pass).
3. Compare draft probabilities to target probabilities token by token, left to right.
4. Accept each candidate token with probability min(1, p_target(token) / p_draft(token)).
5. On the first rejection, resample that position from an adjusted distribution
   (the residual: max(0, p_target - p_draft), renormalized) and discard everything after it.
6. If all k candidates are accepted, additionally sample one bonus token from the target's
   distribution at the final position — effectively a free extra token.

That acceptance rule is the mathematically important part. It is not a heuristic approximation; it is constructed so that, in expectation over many draws, the accepted tokens are distributed exactly as if you had sampled them from the target model alone. This is what makes speculative decoding “lossless” — you get a speedup with zero quality tradeoff, unlike techniques such as quantization or distillation that explicitly trade some accuracy for efficiency.

A Minimal Implementation

Stripped of production concerns (batching, KV cache management across both models, tokenizer alignment), the core rejection-sampling logic looks like this:

import numpy as np

def speculative_step(draft_probs, target_probs, draft_tokens):
    """
    draft_probs:  list of probability distributions the draft model assigned
                  to each of its own sampled tokens, at each of the k positions.
    target_probs: list of the target model's probability distributions at the
                  same k positions, computed in a single parallel forward pass.
    draft_tokens: the k tokens the draft model actually sampled.
    Returns the accepted token sequence plus one bonus/resampled token.
    """
    accepted = []
    for i, token in enumerate(draft_tokens):
        p_target = target_probs[i][token]
        p_draft = draft_probs[i][token]
        acceptance_prob = min(1.0, p_target / p_draft)
        if np.random.rand() < acceptance_prob:
            accepted.append(token)
            continue
        # Rejected: resample from the residual distribution and stop here.
        residual = np.maximum(0, target_probs[i] - draft_probs[i])
        residual /= residual.sum()
        accepted.append(np.random.choice(len(residual), p=residual))
        return accepted

    # All k candidates accepted: sample one bonus token from the target model.
    bonus = np.random.choice(len(target_probs[-1]), p=target_probs[-1])
    accepted.append(bonus)
    return accepted

The speedup comes entirely from step 2 running as one batched forward pass over k+1 positions instead of k+1 sequential passes — the expensive target model’s memory-bandwidth-bound weight loading happens once per verification round instead of once per token.

What Determines the Speedup

The actual wall-clock gain depends on two things pulling in opposite directions:

• Acceptance rate — how often the draft model’s guesses match what the target model would have produced. A draft model that’s a smaller version of the same family, trained on similar data, tends to agree with the target far more often than an unrelated small model.
• Draft cost relative to target cost — the draft model still has to run k times sequentially, so if it’s not meaningfully cheaper than the target, you’ve just added overhead.

Draft/target pairing	Typical acceptance behavior	Practical implication
Same model family, much smaller draft (e.g. a distilled or early checkpoint)	High agreement on easy, predictable spans (boilerplate, common phrasing)	Best general-purpose choice
Same model, early-exit or self-speculative head	High agreement, no separate model to host	Simplifies deployment, no draft/target vocabulary mismatch
Unrelated small model	Lower agreement, more rejections	Speedup shrinks or can even go negative
n-gram / retrieval-based draft (draft from the prompt itself)	Very high agreement on verbatim or near-verbatim spans	Excellent for tasks with lots of copying, like code editing or RAG-style quoting

Rejections aren’t wasted work in the strict sense — you still get the resampled token for free — but every rejection also throws away the remaining candidates in that batch, so a lower acceptance rate directly reduces the average number of tokens produced per verification round.

Self-Speculative Variants

Hosting two separate models adds real operational overhead: two sets of weights in memory, two KV caches, and the need to keep draft and target model vocabularies aligned so token IDs line up during verification. Self-speculative decoding avoids a second model entirely by having the same network produce cheap draft guesses using an early-exit head partway through its own layers, then verifying with the full forward pass. Approaches in this family (sometimes marketed under names like Medusa-style multi-head decoding, or lookahead decoding that speculates using n-grams drawn from the model’s own recent output) trade some implementation complexity for not needing to manage a second model’s lifecycle in production.

Where It Helps and Where It Doesn’t

Speculative decoding is a latency optimization for a single generation stream, not a throughput optimization for a saturated batch server. If your GPU is already running at high batch sizes serving many concurrent users, the memory-bandwidth headroom that speculative decoding exploits (extra capacity to verify several tokens per pass instead of one) may simply not be there — you’re already compute-bound, not idle-bandwidth-bound. It shines most in:

• Low-concurrency, latency-sensitive interactive use (a single user’s chat session, code-completion-as-you-type).
• Workloads with predictable structure the draft model can latch onto — code, structured output, quoting from a provided document.
• Self-hosted deployments where you control both models and can afford the extra memory for a draft model or an early-exit head.

It matters less, or requires careful capacity planning to actually pay off, in high-throughput batch-serving scenarios where the GPU is already busy verifying many independent requests in parallel and has little slack left to absorb a second model’s sequential drafting cost.

Turning It On in a Self-Hosted Stack

Most modern serving engines expose speculative decoding as a configuration flag rather than something you implement yourself. A vLLM-style setup pairing a large target model with a smaller draft model from the same family might look like this:

# vllm serve config
model: meta-llama/Llama-3.1-70B-Instruct
speculative_config:
  method: draft_model
  model: meta-llama/Llama-3.1-8B-Instruct
  num_speculative_tokens: 5
gpu_memory_utilization: 0.90

The num_speculative_tokens value is the k from the draft-and-verify loop above — the number of tokens the draft model guesses per round before the target model verifies them in one pass. Setting it too high wastes draft-model compute on guesses that will likely be rejected past the first divergence; setting it too low leaves speedup on the table on easy, highly predictable spans. Five is a common starting point, worth tuning against your own traffic rather than assumed.

Because the whole point of the technique is that it should be lossless, the right way to validate a new speculative decoding configuration is not to eyeball output quality — it’s to confirm the distribution is unaffected while latency drops:

def compare_configs(prompts, generate_baseline, generate_speculative):
    latencies_baseline, latencies_spec = [], []
    for prompt in prompts:
        t0 = time.perf_counter()
        out_baseline = generate_baseline(prompt, temperature=0)  # greedy, deterministic
        latencies_baseline.append(time.perf_counter() - t0)

        t0 = time.perf_counter()
        out_spec = generate_speculative(prompt, temperature=0)
        latencies_spec.append(time.perf_counter() - t0)

        assert out_baseline == out_spec, "Greedy outputs must match exactly"

    speedup = sum(latencies_baseline) / sum(latencies_spec)
    print(f"Speedup: {speedup:.2f}x, outputs identical across {len(prompts)} prompts")

At temperature=0 (greedy decoding), a correct speculative decoding implementation should produce byte-identical output to the baseline every time, by construction — any divergence there points to a bug in the acceptance logic or a vocabulary mismatch between draft and target, not an acceptable approximation.

Key Takeaways

• Autoregressive decoding is sequentially bottlenecked: each token waits on the last, so wall-clock latency scales with output length regardless of raw GPU throughput.
• Speculative decoding has a small draft model guess several tokens ahead, then verifies all of them in one parallel forward pass of the target model.
• The acceptance rule (accept with probability min(1, p_target/p_draft), otherwise resample from the residual) guarantees the output distribution exactly matches sampling from the target model alone — this is a lossless speedup, not an approximation.
• Realized speedup depends on acceptance rate (how well the draft matches the target) and on the draft model being meaningfully cheaper to run than the target.
• Self-speculative approaches (early-exit heads, n-gram/lookahead drafting) avoid hosting a second model at the cost of some implementation complexity.
• It is primarily a latency win for single-stream, low-concurrency serving; its benefit shrinks in already-saturated, high-throughput batch serving.
• Choosing a draft model from the same family as the target, or one specialized for the task’s typical structure, matters more for realized speedup than raw draft model size.