Implement Speculative Decoding in vLLM with a Draft Model

Configure vLLM with a draft model, manage memory, and verify when speculative decoding fits a self-hosted inference setup using official guidance.

Implement draft-model speculative decoding in vLLM by putting the settings in one --speculative-config JSON object; the vLLM draft-model guide documents Qwen/Qwen3-0.6B proposing 5 tokens per step, the default shared-vocabulary requirement, the use_heterogeneous_vocab relaxation, and the deprecation of --speculative-model and --num-speculative-tokens. For a self hosted llm setup, confirm compatibility and memory limits first, then compare the speculative server with a baseline rather than treating the example token count as a universal setting.

Prepare the target and draft models

Before launching vLLM, record the inputs that determine whether the configuration is meaningful:

The exact target-model value.
The exact draft-model value.
Whether the two models share a vocabulary or will use use_heterogeneous_vocab.
The prompt and output lengths expected in normal use.
The request set, arrival rate and concurrency you intend to evaluate.

Keep the target and draft roles separate in your notes. A model identifier appearing in the draft configuration does not remove the need to check it against the target configuration and the vLLM version installed on the host.

If your draft model does not share the target model’s vocabulary, record that decision explicitly rather than relying on the default. Likewise, migrate any existing command that still uses the separate model and token-count flags instead of combining both configuration styles.

Implement draft-model speculative decoding in vLLM

The speculative decoding documentation defines the implementation contract: pass every speculative setting in one JSON object through --speculative-config, select the approach with method, and set num_speculative_tokens to the positive number of tokens proposed per step. The same keys are accepted from Python through LLM(..., speculative_config={...}).

Use the documented draft-model configuration with your target-model value:

vllm serve YOUR_VALUE \
  --speculative-config '{"model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5, "method": "draft_model"}'

Replace YOUR_VALUE with the target model accepted by your installation. The JSON object is the documented example; its token count is a configuration value, not a general capacity recommendation.

The same documentation positions speculative decoding for reducing inter-token latency under medium-to-low QPS, memory-bound workloads. It says model-based methods such as draft models provide the greatest latency reduction, while n-gram and suffix decoding provide more modest speedups. It also describes speculative sampling as theoretically lossless only within hardware numeric precision. Token log probabilities are not guaranteed to remain stable, so the same request can produce different output across runs; exact text equality is therefore not a sufficient correctness check.

In a self hosted llm deployment, request rate, output length and memory pressure belong in the launch decision. A configuration that suits an interactive, generation-heavy workload may represent a different tradeoff from one serving repeated short requests.

Separate prefix caching from speculative decoding

Automatic Prefix Caching (APC) is a separate mechanism. The Automatic Prefix Caching documentation explains that APC stores the KV cache from earlier queries so later requests with the same prefix can reuse the shared work. This helps most when several questions reuse one long document or when chat history carries across conversation rounds.

APC shortens the prefill phase, not the generation of new tokens. The documentation also says it provides no performance gain when generation dominates because answers are long, or when a new query shares no prefix with earlier queries.

That distinction matters during evaluation. A prefix-cache hit can reduce repeated prefill work without demonstrating anything about draft-model speculation. Keep APC-heavy requests and speculative-decoding measurements separate so one mechanism does not obscure the other.

Set the memory and batching controls

The vllm serve CLI reference defines the principal allocation controls. --gpu-memory-utilization is the GPU-memory fraction assigned to the model executor; its documented default is 0.92, and the limit applies separately to each vLLM instance. --kv-cache-memory-bytes sets KV-cache memory directly per GPU. When it is set, it overrides --gpu-memory-utilization, so decide which allocation mechanism you actually want to control.

Context and batching settings determine the other limits:

--max-model-len covers both prompt and output. If omitted, vLLM derives the length from the model configuration.
--max-num-seqs caps the sequences processed in one iteration.
--max-num-batched-tokens caps the tokens processed in one iteration.

Set the context and batch limits from the workload you need to serve, not from the draft model’s example token count. Then choose an allocation approach deliberately. If you use --kv-cache-memory-bytes, do not assume that changing --gpu-memory-utilization will further adjust KV-cache memory, because the direct byte setting takes precedence.

Check the running server with the benchmark CLI

After starting the server, use the documented online-benchmark form:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

The command above is the documented example; adapt the model identity when applying it to the model served by your instance. The benchmark CLI documentation shows that its report includes request throughput, output-token throughput and total-token throughput. Time to first token (TTFT) is the interval from sending a request to receiving its first streamed output. Inter-token latency (ITL) records gaps between consecutive streamed outputs, while time per output token (TPOT) is calculated once per request, excluding the first token, and then aggregated.

With speculative decoding, one streamed output can contain several tokens. ITL does not add zero-duration gaps for tokens arriving in the same output, so ITL and TPOT should not be treated as interchangeable. Metric terminology is also not standardized across benchmarking tools, so compare like-for-like reports from the same tool and configuration.

--request-rate controls the target request generation rate. Its default, inf, sends all requests immediately; use a finite value for controlled arrival patterns. --max-concurrency can cap outstanding requests, while leaving it unset makes concurrency unlimited.

Compare the baseline and speculative servers with the same dataset, request rate and concurrency settings. Pay particular attention to inter-token latency for the workload described in the vLLM guidance, but retain TTFT, TPOT and throughput as separate signals. Re-running the benchmark against the same server can reuse prompts retained in the prefix cache and inflate throughput, so a repeated run must not be treated automatically as an independent comparison.

Where the implementation usually goes wrong

Copying the example token count without evaluation. The value 5 belongs to the documented example. Your prompt lengths, output lengths and traffic pattern determine whether it is appropriate.
Mixing current and deprecated syntax. Put draft settings in the documented JSON object rather than combining them with the older separate flags.
Treating APC reuse as a speculative result. APC affects repeated prefill work; speculative decoding concerns the generation path.
Changing allocation and speculation together. Keep the baseline stable if you want to attribute a latency change to draft-model speculation rather than to a simultaneous memory or batching change.
Treating different text as proof of invalid sampling. The documented logprob instability means identical requests can differ across runs.
Comparing different arrival conditions. Keep the request set, request rate, concurrency and benchmark tool aligned between baseline and speculative runs.

These steps are compiled from the official documentation linked in the text; no commands or benchmarks were run for this guide, and the steps should be checked against the current version before use.