Capture CUDA Graphs in vLLM Serving

For a self hosted llm, CUDA graph capture is normally already active: vLLM’s default graph mode is FULL_AND_PIECEWISE, a mode the CUDA Graphs design documentation describes as generally the most performant but also the most memory-intensive and slowest to capture. The default -O2 optimization level includes these graphs; if you use --enforce-eager as a diagnostic, the optimization guide says it skips both compilation and graph capture for the fastest startup, at the cost of steady-state decode performance. Start from the normal vllm serve path, retain that path for the capture run, and validate the same application request and a controlled load before and after any configuration change.

Configure a self hosted llm to retain CUDA graphs

Before editing the service, preserve a known-working baseline:

• The complete vllm serve invocation, including all model and deployment arguments.
• The current context, sequence, batch-token, and GPU-memory settings.
• Several representative application requests and their acceptance criteria.
• The client configuration, including its endpoint and decoding options.
• The current startup log and a comparable load-test procedure.

The Conserving Memory guide states that CUDA graphs are enabled by default but take extra GPU memory; it also identifies max_model_len and max_num_seqs as limits you can lower to reduce memory use, and enforce_eager as the setting that disables graph capture completely.

Treat the default FULL_AND_PIECEWISE mode as your baseline rather than changing several controls together. If memory is constrained, change one limit at a time and retain enough headroom for the model, KV cache, and graph memory. Otherwise, a later benchmark result cannot show which change affected serving behavior.

Start the serving process with explicit limits

Do not replace a working deployment with a guessed minimal command. Use the following as a template, inserting the model value and limits from your existing configuration:

vllm serve YOUR_VALUE \
  --gpu-memory-utilization YOUR_VALUE \
  --max-model-len YOUR_VALUE \
  --max-num-seqs YOUR_VALUE \
  --max-num-batched-tokens YOUR_VALUE

The vllm serve CLI reference defines --gpu-memory-utilization as the GPU-memory fraction used by the model executor, with a documented range of 0 to 1, a default of 0.92, and a scope limited to the current vLLM instance; it defines --max-model-len as the context length covering prompt plus output, derived from the model configuration when omitted, while --max-num-seqs caps sequences processed in one iteration and --max-num-batched-tokens caps tokens processed in one iteration.

Fill each YOUR_VALUE deliberately:

• Put the model value from your working command in the first position.
• Use your deployment’s intended GPU-memory fraction rather than copying the documented default blindly.
• Keep the context limit compatible with the longest prompt-and-output combination your application accepts.
• Set sequence and batch-token limits according to the concurrency you actually intend to serve.
• Omit an option when your existing service intentionally relies on its current default.

Do not add --enforce-eager to the captured configuration. If you want an eager comparison, copy the entire command, change only that one setting, and keep the two versions clearly separated. This prevents startup differences from being confused with steady-state serving differences.

If your installed release requires additional arguments or exposes a different graph-mode control, check its current CLI help before starting. Do not copy an unverified flag from another vLLM release.

Check that serving and output still work

Begin with correctness checks rather than throughput. Send a representative request through the same client path used by the application, then record the HTTP result, returned content, required structured fields, and any application-level errors. Repeat the same request after restarting the process with the captured configuration. Use your own acceptance criteria to decide whether the response is correct; a successful response alone does not establish semantic correctness.

Keep the correctness sample representative of production traffic. Include the longest context your service permits if that is a supported use case, but do not mix an unusually long test into a later performance comparison. Run the same samples against the captured and eager configurations.

The Parallelism and Scaling guide explains that the startup-log line GPU KV cache size reports the total number of tokens that can be stored in the GPU KV cache at once. Record that line when diagnosing capacity, but do not treat it as proof that graph capture or output correctness has been verified. Application requests remain the correctness check; the log line provides deployment-capacity context.

A practical sequence is:

1. Start the server with the normal captured configuration.
2. Confirm that startup completes and inspect the cache-capacity line.
3. Send representative requests through the production client path.
4. Compare their results with the pre-change baseline.
5. Restart and repeat to catch process-dependent behavior.
6. Only then begin controlled performance measurements.

Compare captured and eager serving fairly

The Benchmark CLI guide documents ShareGPT for online and offline benchmarks; it defines time to first token as client time from request submission to the first streamed output, inter-token latency as the gaps between streamed outputs, and time per output token as per-request timing excluding the first token; it also reports request, output-token, and total-token throughput, defaults request rate to inf, treats omitted maximum concurrency as unlimited, and warns that repeated runs against the same server can reuse prefix-cache prompts and inflate throughput.

Download the documented ShareGPT dataset with:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

With the server running, use the documented online benchmark command:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

For a controlled comparison, append the same finite load controls to both configurations:

  --request-rate YOUR_VALUE \
  --max-concurrency YOUR_VALUE

Keep the model, endpoint, dataset, client host, context limit, sequence limit, batch-token limit, request rate, and maximum concurrency aligned between runs. Note whether each run used a newly started server; a repeated run against the same warm server may benefit from retained prefix-cache prompts.

Record time to first token, inter-token latency, time per output token, request throughput, output-token throughput, and total-token throughput. Do not declare a captured or eager configuration the winner until both reports used comparable load and server state. Performance results and application-output correctness should remain separate acceptance decisions.

Where CUDA graph checks usually go wrong

Use this checklist when results are inconsistent:

• Eager mode is hidden in a wrapper. Inspect the final process arguments for --enforce-eager before labeling a run as captured.
• The memory budget changed. Restore the model and runtime limits to the same values before comparing configurations; investigate memory tuning as a separate experiment.
• Several controls moved together. A change to context length, sequence limits, or batched tokens makes it difficult to identify the source of a serving difference.
• Startup time was compared with decode behavior. Record startup and steady-state serving in separate sections of the report.
• The benchmark reused a warm prefix. Prefer a clean server lifecycle for each condition when possible, and record whether the server was reused.
• The cache-capacity line was treated as correctness. Use it to understand KV-cache capacity, then validate returned content through representative application requests.

These steps are compiled from the official documentation linked in the text; no commands were run for this guide, and they should be checked against the current version before use.

Sources

• CUDA Graphs — vLLM design docs
• Optimization and Tuning — vLLM docs
• Conserving Memory — vLLM docs
• vllm serve — CLI Reference
• Parallelism and Scaling — vLLM docs
• Benchmark CLI — vLLM docs