Tune vLLM Block Size for Throughput and VRAM

Tune vLLM block size for throughput and VRAM by checking the current release first: the official vLLM vllm serve CLI reference does not identify a block-size engine argument, so there is no documented universal value to apply here. For a self hosted llm, change a documented cache or scheduler setting at a time, hold the workload constant, and compare cache capacity, throughput, latency, and observed VRAM.

Questions self hosted llm operators should ask about block size

Automatic Prefix Caching (APC) is where vLLM’s documentation uses KV-cache blocks. The Automatic Prefix Caching design hashes each block using its tokens and the prefix tokens before it, caches only full blocks, and can include LoRA IDs, multimodal input hashes, and cache salts in block identity. These rules determine cache eligibility and identity, but they do not define a tunable number of tokens per block.

Do not confuse block hashing with block size. Since v0.11, the documented prefix-cache hash setting has used sha256 by default, while sha256_cbor provides reproducible cross-language hashes:

--prefix-caching-hash-algo sha256_cbor

That option selects a hash algorithm; it does not set the number of tokens in a block. If your installed vLLM release exposes a block-size control that is absent from the linked documentation, use the exact name, allowed values, and compatibility conditions in that release’s current project documentation. Until those are documented, adding an unsupported flag would not be a defensible tuning procedure.

Prepare a controlled vLLM baseline

Record the exact vLLM version, model ID, GPU count, parallel settings, optimization mode, prompt and output limits, dataset, and endpoint. Keep these constant while changing a single cache or scheduler setting.

The CLI definitions for vllm serve define --max-model-len as the combined prompt-and-output context length, --max-num-seqs as the sequence limit for an iteration, and --max-num-batched-tokens as the token limit for an iteration. They also define --gpu-memory-utilization as a fraction from 0 to 1; its documented default is 0.92, and the limit applies per vLLM instance. If --max-model-len is omitted, vLLM derives it from the model configuration.

Use a template such as:

CUDA_VISIBLE_DEVICES=YOUR_VALUE vllm serve YOUR_VALUE \
  --max-model-len YOUR_VALUE \
  --max-num-seqs YOUR_VALUE \
  --max-num-batched-tokens YOUR_VALUE \
  --gpu-memory-utilization YOUR_VALUE

Replace each YOUR_VALUE according to its position. The Conserving Memory guide says to select devices through CUDA_VISIBLE_DEVICES rather than with PyTorch device functions before vLLM starts. It also notes that every tensor-parallel process reads the whole checkpoint, making disk-loading time grow with tensor-parallel size. Keep device selection and parallel configuration fixed during the comparison.

Tune throughput with supported cache controls

The Conserving Memory guide identifies max_model_len and max_num_seqs as controls that can reduce memory usage when lowered. Treat them as workload constraints: an unnecessarily long context or sequence limit can leave less room for cache and concurrent work.

The Optimization and Tuning guide gives a direct batching trade-off: smaller max_num_batched_tokens values, with 2048 given as an example, can improve inter-token latency by reducing prefills that slow decoding, while higher values can improve time to first token by processing more prefill tokens in a batch. For throughput, the guide recommends max_num_batched_tokens > 8192, especially for smaller models on large GPUs. These are documented tuning directions, not guaranteed results for your model or workload.

Candidate	Change from baseline	Primary measurement
Baseline	No configuration change	Reference result
Memory	Lower max_model_len or max_num_seqs	Observed VRAM and cache capacity
Decode	Lower max_num_batched_tokens	Inter-token latency
Prefill or throughput	Raise max_num_batched_tokens	Time to first token and throughput

Avoid changing context length, sequence limits, and batched-token limits together if you need to identify which setting produced the change.

Run the same load for each candidate

The Benchmark CLI’s ShareGPT and caching notes list ShareGPT for online and offline benchmarks. They also warn that repeating vllm bench serve against the same server can reuse prompts left in the prefix cache and inflate throughput. Download the documented dataset with:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

For controlled online runs, the benchmark options and metric definitions state that --request-rate controls requests per second; finite values simulate controlled load, while inf means maximum throughput and is the default that sends requests immediately. The same report provides request throughput, output-token throughput, and total-token throughput; time to first token runs from request submission to the first streamed output, inter-token latency records gaps between streamed outputs, and time per output token is calculated once per request excluding the first token.

vllm bench serve \
  --backend vllm \
  --model YOUR_VALUE \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --request-rate YOUR_VALUE

Fill the request rate with the same finite value for every candidate. Keep the model, endpoint, dataset, and benchmark command unchanged. Label results as cache-assisted or cache-cold rather than treating an immediate repeat against the same server as an independent result.

Check VRAM without confusing cache capacity

At startup, the Parallelism and Scaling guide says the GPU KV cache size line reports how many tokens fit in the GPU KV cache at once. Record that line for every candidate.

This startup value is cache capacity expressed in tokens, not observed peak VRAM expressed in bytes. The benchmark instructions do not specify a VRAM-monitoring command. Use the monitoring utility prescribed by your GPU platform, document its sampling method, and record peak VRAM separately.

Draw a throughput conclusion only when request rate, dataset, model, endpoint, and cache state are comparable. Draw a VRAM conclusion only when the same monitoring method is used. If KV-cache capacity changes but observed VRAM does not, report both results rather than presenting token capacity as a VRAM measurement.

Where block-size tuning usually goes wrong

The Optimization and Tuning guide says KV-cache exhaustion can cause request preemption and later recomputation. It lists increasing gpu_memory_utilization as the first remedy for frequent preemption, or lowering max_num_seqs or max_num_batched_tokens to reduce concurrent requests and KV-cache demand. It also says vLLM V1 defaults to RECOMPUTE, and that when chunked prefill is disabled, max_num_batched_tokens must be greater than max_model_len.

The same guide describes --enforce-eager as skipping compilation and CUDA-graph capture for the fastest startup, at the cost of steady-state decode performance. Keep that option unchanged across candidates; otherwise the benchmark is comparing execution modes as well as cache settings.

The best direction depends on the model, prompt and output mix, prefix reuse, hardware, and current vLLM release. Treat preemption, cache capacity, latency, throughput, and observed VRAM as separate signals rather than assuming that changing an undocumented block-size value will improve them together.

No commands or benchmarks were run for this article. These steps are compiled from the official documentation linked in the text and should be checked against the current vLLM version before use.

Sources

• vLLM vllm serve CLI Reference
• vLLM Automatic Prefix Caching Design
• vLLM Conserving Memory
• vLLM Optimization and Tuning
• vLLM Benchmark CLI
• vLLM Parallelism and Scaling