AI DeploySelf-hosted LLMGet it deployed

Configuring vLLM Request Queues and Continuous Batching

Learn how vLLM request and token limits, chunked prefill, KV cache controls, and repeatable load checks shape reliable deployments.

For a self hosted llm, configure queue pressure with max_num_seqs, max_num_batched_tokens, and max_model_len; the vLLM vllm serve CLI reference defines these as the sequence cap for one iteration, token cap for one iteration, and model context covering prompt plus output. Set gpu_memory_utilization to the fraction intended for the model executor, then treat continuous batching as a workload budget that must be checked under client-like load rather than copied from unrelated hardware.

Prepare the queue budget

Before launching vLLM, write down the longest prompt plus expected output, the peak number of simultaneous sequences, and whether the main objective is time to first token, inter-token latency, or throughput. Keep client timeouts, retries, and backoff policies in your API gateway or application; the batching controls discussed here define scheduler limits rather than client retry behavior.

The vLLM SchedulerConfig source lists defaults of 2048 for max_num_batched_tokens and 128 for max_num_seqs. These are source defaults, not guaranteed capacity or performance targets. Your context envelope, concurrency pattern, and GPU memory determine whether they are appropriate.

Configure the request queue

Build the online-serving command from the documented engine arguments. Replace each YOUR_VALUE from your workload and current model documentation:

MODEL=YOUR_VALUE

vllm serve "$MODEL" \
  --max-model-len YOUR_VALUE \
  --max-num-seqs YOUR_VALUE \
  --max-num-batched-tokens YOUR_VALUE \
  --gpu-memory-utilization YOUR_VALUE

Set max_model_len to cover both the prompt and generated output. Use max_num_seqs as the concurrency boundary for a single iteration and max_num_batched_tokens as its token-volume boundary. If you omit gpu_memory_utilization, the CLI reference documents a default of 0.92; the setting applies per vLLM instance rather than globally to the GPU.

Avoid raising every limit together. A larger context allowance, sequence cap, and token cap can place different demands on memory even when the application sends similar requests. Establish the workload envelope first, then vary the queue settings for a specific objective.

Tune continuous batching for a self hosted llm

The vLLM Optimization and Tuning guide states that V1 enables chunked prefill by default whenever possible and prioritizes decode requests when it is active. This makes max_num_batched_tokens a scheduling trade-off rather than a universally larger-is-better setting.

The guide gives 2048 as an example of a smaller value that can improve inter-token latency because fewer prefills interrupt decoding. Higher values can improve time to first token because more prefill tokens can be processed together. For throughput, the guide recommends setting max_num_batched_tokens > 8192, especially for smaller models on large GPUs. These are documented tuning directions, not benchmark results from this article.

The scheduler constraint also depends on chunked prefill. When chunked prefill is disabled, max_num_batched_tokens must be greater than max_model_len. Keep that relationship visible in configuration management rather than changing the values independently.

KV-cache exhaustion can cause vLLM to preempt requests, after which preempted requests are recomputed when sufficient cache space becomes available. In V1, RECOMPUTE is the default preemption mode rather than SWAP. If preemption is frequent, the documented remedy order is to increase gpu_memory_utilization first, then decrease max_num_seqs or max_num_batched_tokens to reduce concurrent cache demand.

Protect KV cache while tuning

The vLLM memory-conservation guide identifies max_model_len and max_num_seqs as ways to reduce memory use. CUDA graphs are enabled by default and consume additional GPU memory. The same guide says --enforce-eager disables graph capture completely.

Treat --enforce-eager as an explicit execution-mode decision, not as a routine queue fix. First align context length and concurrency with the requests your clients actually send. Then consider changing graph behavior if memory remains the constraint and you have reviewed its effect on the current release.

Check the server endpoints

The vLLM Online Serving guide lists /version for version information, /load for server-load metrics, /v1/models for the available models, and /health for a health check.

Set BASE_URL to the address used when starting the server:

BASE_URL=YOUR_VALUE

curl "$BASE_URL/version"
curl "$BASE_URL/load"
curl "$BASE_URL/v1/models"
curl "$BASE_URL/health"

These checks confirm that the documented endpoints respond. They do not establish that continuous batching is well tuned, because they do not create the request pattern needed to expose queue, latency, or KV-cache problems.

Test queue behavior under load

The vLLM Benchmark CLI guide positions vllm bench serve for feature evaluation and regression testing. Its documented online command points the benchmark at a running server:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

ShareGPT is supported for online and offline benchmarks and is downloaded from Hugging Face. The --request-rate option defaults to inf, which sends all requests immediately for maximum-throughput testing. Add a finite --request-rate YOUR_VALUE when you want controlled request generation.

The report includes request throughput, output-token throughput, and total-token throughput. Time to first token, or TTFT, runs from sending a request to receiving its first streamed output. Inter-token latency, or ITL, records gaps between streamed outputs. Time per output token, or TPOT, is calculated once per request while excluding the first token.

Metric terminology is not standardized across benchmarking tools, so retain the tool name and configuration with every result. Repeating vllm bench serve against the same server can reuse prompts in the prefix cache and inflate throughput. Account for that cache reuse before interpreting a rerun as a configuration improvement.

For a custom load shape, the same guide requires a JSONL dataset with a prompt field for each entry and selects it with --dataset-name custom.

Fix common configuration failures

  • Context-limit failures: Compare the complete prompt plus requested output with max_model_len. Reduce the application envelope or set a different explicit context boundary rather than assuming max_num_seqs controls input length.
  • Conflicting latency goals: Smaller token budgets can reduce prefill interference during decoding, while larger budgets can process more prefill tokens together. Select the direction that matches the workload rather than optimizing TTFT and ITL as if they were identical.
  • Frequent preemption: Treat it as KV-cache pressure. Apply the documented order: adjust gpu_memory_utilization, then reduce max_num_seqs or max_num_batched_tokens if necessary.
  • Unexpected memory pressure: Recheck context length, concurrency, and CUDA graph memory before changing the execution mode.
  • A suspiciously strong benchmark rerun: Check for prefix-cache reuse before attributing the change to queue settings.

These steps are compiled from the official documentation linked in the text; no commands were run for this article, and the settings should be checked against the current version before use.

Sources