AI DeploySelf-hosted LLMGet it deployed

Tune vLLM Batching, PagedAttention, and Quantization

Configure vLLM batching and KV-cache limits, check PagedAttention capacity, and choose quantization only for hardware supported by current documentation.

For ai deployment with vLLM, tune batching through the documented sequence and token limits, check KV-cache capacity when assessing a PagedAttention setup, and choose quantization only after checking hardware support. The documentation used here does not specify a standalone PagedAttention engine argument, so check your installed version’s current engine-argument reference before using one. Keep capacity, precision, and latency decisions separate instead of assuming one setting improves every objective.

Prepare an ai deployment baseline

Start by recording the model identifier, prompt and output-length distribution, expected concurrency, target GPU configuration, and whether the main objective is time to first token, inter-token latency, or throughput. Keep the complete startup command with the model revision; otherwise, a later comparison may mix model, hardware, and scheduler changes.

The vllm serve CLI reference defines --max-model-len as the context length covering prompt and output and says an omitted value is derived from the model configuration; it defines --max-num-seqs as the maximum sequences processed in one iteration, --max-num-batched-tokens as the maximum tokens processed in one iteration, both parallelism sizes as defaulting to 1, and --gpu-memory-utilization as a 0 to 1 per-instance fraction with a documented default of 0.92.

Use those documented options to preserve a repeatable baseline:

vllm serve YOUR_VALUE \
  --tensor-parallel-size YOUR_VALUE \
  --pipeline-parallel-size YOUR_VALUE \
  --max-model-len YOUR_VALUE \
  --max-num-seqs YOUR_VALUE \
  --max-num-batched-tokens YOUR_VALUE \
  --gpu-memory-utilization YOUR_VALUE

Omit parallelism options when the model does not require them. Fill the remaining placeholders from your model and workload rather than copying values from an unrelated deployment.

Tune batching for the workload

According to the Optimization and Tuning guide, vLLM V1 enables chunked prefill by default whenever possible and prioritizes decode requests; it gives 2048 as an example of a smaller max_num_batched_tokens value for better inter-token latency because fewer prefills slow decodes, says larger values improve time to first token by processing more prefill tokens, recommends a value above 8192 for optimal throughput—especially for smaller models on large GPUs—and requires max_num_batched_tokens to be greater than max_model_len when chunked prefill is disabled; the same guide explains that preemption occurs when KV-cache space runs out, that preempted requests are recomputed later, that V1 defaults to RECOMPUTE rather than SWAP, and that the documented remedies are to raise gpu_memory_utilization or reduce max_num_seqs or max_num_batched_tokens.

Translate those trade-offs into controlled runs. Keep the sequence cap and token cap distinct, change a single setting at a time, and retain the same prompts and arrival pattern. If first-token responsiveness is the priority, test larger token batches; if smooth decoding is the priority, test smaller budgets. These are tuning directions, not universal defaults.

PagedAttention: verify KV-cache capacity

Treat PagedAttention here as a KV-cache capacity question, not as a hunt for an undocumented switch. The concrete startup signal is GPU KV cache size: the Parallelism and Scaling guide says this line reports the total number of tokens that the GPU KV cache can store at once.

Compare that capacity with the context length and concurrency you intend to serve. It is a capacity indicator, not a latency or throughput result. If the available cache cannot support the planned workload, revisit the documented context, sequence, token-batch, and GPU-memory controls before changing anything else.

Frequent preemption is a separate warning sign. Use the scheduler remedies cited above as a troubleshooting path, but do not treat them as guaranteed improvements or copy their values without checking the memory available to your instance.

Choose quantization against the hardware

The vLLM quantization guide describes quantization as trading model precision for a smaller memory footprint and identifies LLM Compressor as a starting point supporting FP8, INT8, INT4, and other formats. Its documented hardware matrix records the following support:

Format Documented hardware support
AWQ Not supported on Volta or AMD GPUs; supported on Turing, Ampere, Ada, Hopper, Intel GPU, and x86 CPU
llm-compressor FP8 (W8A8) Supported on Ada, Hopper, and the listed AMD GPU columns
GGUF Supported on NVIDIA Volta through Hopper and AMD GPU; not supported on Intel GPU or CPUs
bitsandbytes Supported on NVIDIA Volta through Hopper

The matrix maps Volta to SM 7.0, Turing to 7.5, Ampere to 8.0 or 8.6, Ada to 8.9, and Hopper to 9.0. It also warns that compatibility can change as vLLM expands support, so check the matrix again when upgrading.

For an FP8 workflow, the FP8 W8A8 guide documents pip install llmcompressor, separate environments for vLLM and llm-compressor, hardware acceleration on GPUs such as NVIDIA H100 and AMD MI300x, NVIDIA FP8 computation at compute capability 8.9 or higher on Ada Lovelace, Hopper, or Blackwell, and W8A16 weight-only operation through Marlin kernels on Turing and Ampere.

The documented dynamic recipe uses these exact settings:

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

This is the recipe configuration, not a complete model-loading and saving script. The guide says this FP8_DYNAMIC flow requires no calibration data because weight quantization uses simple RTN and activations are quantized dynamically. Use the guide’s complete current recipe for the surrounding model and save operations.

Check the result with vllm bench serve

After the server starts, inspect GPU KV cache size, then run the same benchmark against the running endpoint:

vllm bench serve \
  --backend vllm \
  --model YOUR_VALUE \
  --endpoint /v1/completions \
  --dataset-name sharegpt

The Benchmark CLI guide says the report includes request throughput and output-token and total-token throughput; defines TTFT as the interval from sending a request to receiving its first streamed output, ITL as the interval between consecutive streamed outputs, and TPOT as a per-request calculation excluding the first token that is then aggregated; documents --request-rate as requests per second, with inf meaning maximum-throughput testing and finite values providing controlled load; and warns that repeating the benchmark against the same server can reuse prompts in its prefix cache and inflate throughput.

Keep the model, dataset, request rate, and workload fixed when comparing configurations. Record request throughput separately from TTFT, ITL, and TPOT rather than combining them into one conclusion. Benchmark terminology is not standardized across tools, so preserve the tool name and metric definition with each result.

Where the tuning usually goes wrong

  • Treating max_num_seqs as a token budget or max_num_batched_tokens as a request count.
  • Changing context length, concurrency, GPU-memory utilization, and quantization together, leaving no clear source of improvement or regression.
  • Treating the GPU KV cache size line as a performance result rather than a capacity constraint.
  • Selecting AWQ, FP8, GGUF, or bitsandbytes from the model label without checking the documented hardware matrix.
  • Comparing repeated benchmark runs without accounting for prefix-cache reuse and request-rate differences.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use. No command or model was run or measured for this article.

Sources