Speculative Decoding Explained: How to Get 2–3× Faster LLM Inference at Zero Quality Loss

Speculative decoding is one of the few LLM optimizations that gives you real latency wins without touching output quality. Instead of the target model generating one token per forward pass, a smaller draft model proposes several tokens ahead, and the target model verifies them in a single parallel pass. Accepted tokens are kept; rejected ones fall back to normal decoding. The math guarantees the output distribution is identical to the target model running alone.

Why it works

Decode is memory-bandwidth-bound. A single forward pass of a 70B model spends most of its time streaming weights from VRAM — the actual compute is nearly free. Verifying 4 draft tokens in one pass costs almost the same wall-clock time as generating 1 token normally. If 3 of the 4 are accepted, you've effectively 3×'d throughput per pass.

The three moving parts

• Draft model. Usually 10–20× smaller than the target (e.g., a 7B drafting for a 70B). Must share the tokenizer. Distilled drafts trained on the target's outputs raise acceptance rates.
• Speculation length (γ). How many tokens to draft per round. Too short wastes verification bandwidth; too long wastes rejected drafts. Typical values: 4–8.
• Acceptance rate. The percentage of drafted tokens that survive verification. 60–80% is normal for well-matched drafts on in-domain text; it collapses on out-of-distribution prompts (rare code, non-English, long tool-call JSON).

Variants worth knowing

Medusa replaces the separate draft model with extra prediction heads on the target itself — no second model to serve, but requires fine-tuning. EAGLE drafts in feature space rather than token space, pushing acceptance rates above 80%. Lookahead decoding removes the draft model entirely by using n-gram parallelism from the target's own history.

Where it stacks (and where it doesn't)

Speculative decoding composes cleanly with FP8 quantization, paged attention, and continuous batching — the gains multiply. It does not help with prefill (already parallel), and it can hurt throughput under heavy batching because verification consumes GPU cycles that could serve more concurrent decode steps. The right regime is latency-sensitive, low-to-medium concurrency workloads.

Quick decision guide

For teams who would rather ship features than tune drafters, SNAP Capture and the broader Silverberry stack handle the serving optimizations under the hood.