Tune vLLM Memory for 128K-Token Contexts

Configure a 128K context in vLLM by setting context, cache, batching, and parallelism limits when the model and GPU support it.

Treat a 128K context as a memory budget, not a switch: --max-model-len covers prompt plus output and otherwise comes from the model configuration, as specified in the vLLM serving CLI reference. For self hosted llm deployments, tune model placement, KV-cache allocation, and batching, then verify that the full target fits.

Set a self hosted llm memory budget

Start with a workload envelope rather than a GPU count. Record the longest prompt, desired output, expected number of active sequences, and whether the priority is time to first token, inter-token latency, or throughput. Treat the 128K context target, concurrency target, and batched-token target as separate decisions: lowering one does not prove that the others will fit.

For a multi-node deployment, the vLLM parallelism and scaling guide sets tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes. When GPUs in a node lack NVLINK, with L40S given as an example, the guide recommends pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead.

Start with a bounded serving command

The Conserving Memory guide directs you to control device selection with CUDA_VISIBLE_DEVICES rather than calling PyTorch device functions before vLLM starts; it also notes that every tensor-parallel process reads the whole checkpoint, making disk-loading time grow with tensor-parallel size.

The vLLM serve CLI reference documents that engine arguments are passed to vllm serve, that --gpu-memory-utilization is the per-instance 0-to-1 fraction used by the model executor and defaults to 0.92, that --max-num-seqs and --max-num-batched-tokens cap sequences and tokens processed in one iteration, and that the tensor- and pipeline-parallel flags default to 1.

Use a bounded template rather than copying settings from unrelated hardware:

export CUDA_VISIBLE_DEVICES=YOUR_GPU_IDS

vllm serve YOUR_MODEL \
  --max-model-len 128k \
  --gpu-memory-utilization YOUR_GPU_MEMORY_UTILIZATION \
  --max-num-seqs YOUR_MAX_NUM_SEQS \
  --max-num-batched-tokens YOUR_MAX_NUM_BATCHED_TOKENS \
  --tensor-parallel-size YOUR_TENSOR_PARALLEL_SIZE \
  --pipeline-parallel-size YOUR_PIPELINE_PARALLEL_SIZE

Replace every placeholder before starting the server. For a single-node tensor-parallel layout, leave pipeline parallelism at its documented default of 1; for a multi-node layout, apply the node-and-GPU sizing rule above.

Reduce KV-cache and runtime memory

The Conserving Memory guide identifies max_model_len and max_num_seqs as direct memory controls, and notes that CUDA graphs are enabled by default while consuming extra GPU memory. If 128K is non-negotiable, preserve --max-model-len 128k and reduce concurrency instead. You can also use --enforce-eager to turn CUDA graph capture off entirely when isolating graph-related memory use.

The Optimization and Tuning guide says decreasing max_num_seqs or max_num_batched_tokens reduces concurrent requests in a batch and therefore reduces KV-cache demand. It also explains that V1 defaults to RECOMPUTE preemption rather than SWAP, with preempted requests recomputed when sufficient cache space becomes available. For frequent preemption, its first remedy is to increase gpu_memory_utilization; reducing sequence or batched-token limits is the second.

The same optimization guidance says chunked prefill is enabled by default whenever possible in V1 and that scheduling prioritizes decode requests. It gives directional batch-token tradeoffs rather than a universal 128K setting: a smaller value such as 2048 can improve inter-token latency, larger values can improve time to first token, and the throughput guidance recommends max_num_batched_tokens > 8192 for smaller models on large GPUs. If chunked prefill is disabled, max_num_batched_tokens must be greater than max_model_len.

Quantize only after checking hardware support

Quantization can reduce the model’s memory footprint, but it does not prove that the current GPU allocation can hold a 128K KV cache. The vLLM quantization guide describes the format as a trade-off between model precision and memory use, while warning that its hardware compatibility chart changes as support evolves.

Check the exact quantization format and hardware entry before changing the model. If the chart does not list your combination, do not assume that a format name alone is sufficient. After any quantization change, repeat the startup capacity check rather than carrying forward a previous GPU KV cache size result.

Check whether 128K actually fits

At startup, read the GPU KV cache size line. The parallelism and scaling guide defines it as the total number of tokens that can be stored in the GPU KV cache at once.

If the reported capacity is below 128K, the current configuration cannot store the full target context at once. Revisit GPU-memory utilization, sequence limits, batched-token limits, and the tensor- or pipeline-parallel layout. If capacity is sufficient, that clears the cache-capacity check, but you still need a workload test with representative prompt and output lengths.

Benchmark the endpoint without confusing the result

The Benchmark CLI guide documents the ShareGPT download and the running-server command shown below. It defines time to first token as client time from sending a request to receiving its first streamed output, inter-token latency as the gaps between streamed outputs, and time per output token as a per-request calculation excluding the first token; the report also lists request, output-token, and total-token throughput.

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

vllm bench serve \
  --backend vllm \
  --model YOUR_MODEL \
  --endpoint /v1/completions \
  --dataset-name sharegpt

That command alone does not establish performance for your specific 128K workload. The same Benchmark CLI guide accepts custom datasets as JSONL with a prompt field selected through --dataset-name custom, and warns that repeated runs against the same server can reuse prompts in the prefix cache and inflate throughput.

Prepare representative long-context entries before benchmarking:

{"prompt":"YOUR_128K_PROMPT"}

Replace the placeholder with the actual prompt text. Keep the server configuration fixed while comparing results, and do not treat repeated same-server runs as independent workload samples.

Avoid the usual 128K tuning errors

Confirm that the model configuration permits the intended prompt-plus-output length.
Choose the parallel layout before increasing batching pressure.
Keep max_model_len at the required target and lower sequence or batch limits first.
Read GPU KV cache size before treating a successful startup as sufficient.
Apply quantization only after checking the current format-and-hardware chart.
Benchmark representative long inputs while accounting for prefix-cache reuse.

These steps are compiled from the official documentation linked in the text; no hands-on run or benchmark was performed for this article, so check them against the current version before use.