AI DeploySelf-hosted LLMGet it deployed

Enable vLLM Prefix Caching for Repeated Prompts

Enable vLLM prefix caching for repeated prompts, reuse shared KV-cache blocks, and verify prefill-focused performance in your own workload.

For llm hosting with repeated prompts, set enable_prefix_caching=True in the Python vLLM engine; the same documentation says APC reuses KV-cache data from earlier queries with the same prefix, skips shared prefill computation, leaves token generation unchanged, is useful for repeated questions over one document or reused chat history, and gives no performance gain when answers dominate or new queries share no prefix. Treat it as a prefill optimization to evaluate against your own traffic, not as a promise of lower end-to-end latency.

Prepare the repeated-prompt workload

The current CacheConfig source declares enable_prefix_caching: bool = True. An omitted setting may therefore inherit the enabled default, so record True explicitly in your deployment configuration and do not use an omitted key as an APC-disabled baseline.

Before editing the service, identify what actually repeats:

  • Find the stable leading token sequence, such as fixed instructions, the same document section, or retained chat history.
  • Prepare a shared-prefix dataset and a comparison dataset whose leading content differs.
  • Keep the model, endpoint, generation settings, request rate, concurrency, and traffic pattern unchanged during the comparison.
  • Decide in advance that prefill-oriented latency is the primary signal, while still retaining decoding metrics as a control.

Request-specific text should follow the stable prefix. Moving an identifier, timestamp, or user-specific value to the beginning can prevent the later content from sharing the intended prefix.

Enable vLLM prefix caching

The Engine Arguments reference states that engine arguments are passed to LLM(...) for offline use and to vllm serve for online serving.

For a Python engine, add this exact argument to your existing initialization:

enable_prefix_caching=True

Place it inside the existing LLM(...) call, keep the Boolean unquoted, and preserve the model and generation arguments already required by your deployment. Do not add a second copy of the key or convert its value to a string.

For online serving, the engine-argument reference establishes that vllm serve accepts engine settings, but the APC instruction cited above specifies the Python form rather than a universal command-line spelling. Check your installed release’s current CLI reference for the supported flag instead of assuming that the Python key can be mechanically converted into a command-line option.

Apply the setting through your normal deployment procedure, then retain the effective configuration and vLLM version with the run record. For the first comparison, avoid unrelated changes to memory, batching, parallelism, or generation behavior.

Match the prefix vLLM can reuse

The prefix-caching design explains that vLLM hashes each KV-cache block using both the tokens inside that block and the tokens in the preceding prefix. Only full blocks are cached. Block identity can also include LoRA IDs, multimodal input hashes, and cache salts used to isolate multi-tenant caches. As of v0.11, the documented default hash algorithm is sha256; the design also documents sha256_cbor for reproducible cross-language hashes through --prefix-caching-hash-algo.

Those rules determine what a useful repeated-prompt test should look like:

  • Use the same tokens, not merely similar wording. A changed leading instruction can change the prefix identity of every block that follows it.
  • Expect reuse from full blocks. A short shared segment does not guarantee cached reuse simply because the strings are identical.
  • Keep relevant identity inputs stable. Changing a LoRA ID, multimodal input hash, or cache salt can prevent otherwise similar text from matching.
  • Leave the hash algorithm unchanged for ordinary workloads. The cross-language option matters when independently implemented systems must reproduce the same hash.
  • Keep changing data after the shared content. Put stable instructions or document text first and request-specific material afterward.

Check the model architecture before investigating cache misses. The current SchedulerConfig source states that encoder-decoder models do not support chunked prefill or prefix caching and that the scheduler disables both for those models.

Verify the change under your llm hosting load

The Benchmark CLI guide defines a custom dataset as JSONL with a prompt field on every entry and selects it with --dataset-name custom. It defines time to first token as the client-observed interval from sending a request to receiving its first streamed output, inter-token latency as the gap between consecutive streamed outputs, and time per output token as a per-request value that excludes the first token. The guide also warns that repeating a benchmark against the same server can reuse prompts in the prefix cache and inflate throughput.

A minimal custom file can place the same prefix at the beginning of each prompt:

{"prompt":"YOUR_SHARED_PREFIX REQUEST_A"}
{"prompt":"YOUR_SHARED_PREFIX REQUEST_A"}
{"prompt":"YOUR_SHARED_PREFIX REQUEST_B"}

Use a separate comparison file in which the leading content differs. The dataset format and selector are documented, but the appropriate file-path option can change between releases, so consult the current CLI page before assembling the custom command rather than guessing an option.

For command syntax, the guide documents this online example:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

That example confirms the command shape and endpoint path, but it does not establish that the ShareGPT prompts share a prefix. Use the custom dataset for the repeated-prompt comparison.

Keep the evaluation controlled:

  • Run the shared-prefix and non-shared-prefix datasets under the same serving configuration.
  • Hold --request-rate, --max-concurrency, and --burstiness constant.
  • Use streamed responses when evaluating time to first token and inter-token latency.
  • Record whether the server is starting fresh or retaining prompts from an earlier run.
  • Compare the full distributions rather than treating a single request as conclusive.
  • Keep inter-token latency and time per output token beside time to first token so that a prefill change is not mistaken for a decoding change.

The cited reporting details do not define a direct APC cache-hit count. If exact hit-level visibility is required, check the current metrics or logging documentation; do not convert higher throughput into a specific number of cache hits.

Read the result without overclaiming

A reduction in time to first token on the shared-prefix workload, relative to the non-shared workload under the same conditions, is consistent with APC reducing prefill work. It is not, by itself, proof of an exact number of reused blocks.

If similar prompts show no benefit, revisit the stable token prefix, full-block requirement, extra hash inputs, and model architecture. If the second benchmark run shows a sudden throughput increase, do not count it as an independent sample: the official benchmark guidance warns that retained prompts can be reused.

Likewise, unchanged decoding metrics do not contradict a prefill improvement. The phase boundary is central to what APC addresses. Keep conclusions limited to the prompt pattern, serving configuration, model, and vLLM version used for the comparison.

These steps are compiled from the official documentation linked in the text; no commands were run for this article, so check the configuration and benchmark options against the current vLLM version before use.

Sources