AI DeploySelf-hosted LLMGet it deployed

Tune vLLM for a 7B Model on an RTX 4090

Choose vLLM memory and batching settings for a 7B model on an RTX 4090, then verify KV-cache capacity and diagnose common runtime failures.

For hosting llm locally, tune vLLM for a 7B model on an RTX 4090 by setting explicit context and concurrency limits and checking the startup KV-cache capacity before accepting the workload. Official documentation does not guarantee that an unspecified 7B checkpoint fits, so validate the exact model artifact, precision or quantization, prompt-plus-output length, and expected concurrency rather than treating the parameter count as a complete capacity calculation.

Prepare the RTX 4090 and define the workload

NVIDIA's RTX 4090 specifications list 24 GB of GDDR6X memory, 450 W total graphics power, an 850 W required system power figure, and a Founders Edition body measuring 304 mm by 137 mm across three slots. Check case clearance and host power before configuring vLLM.

Before launch, record:

  • The exact model identifier and local model artifact.
  • The maximum prompt-plus-output context you will permit.
  • The largest number of sequences you expect to keep active.
  • Whether the primary goal is inter-token latency, time to first token, or throughput.
  • The quantization format, if any.

The vllm serve CLI reference defines --max-model-len as prompt-plus-output context derived from the model configuration when omitted, --max-num-seqs as the number of sequences processed in one iteration, --max-num-batched-tokens as the number of tokens processed in one iteration, and --gpu-memory-utilization as the model executor's memory fraction; it documents a per-instance default of 0.92, a default tensor-parallel group count of 1, and --dtype auto selecting FP16 for FP32 or FP16 models and BF16 for BF16 models.

Tune vLLM memory for hosting llm locally with a 7B model

The vLLM memory-conservation guide says that lowering max_model_len and max_num_seqs can reduce memory use, that CUDA graphs are enabled by default and consume extra GPU memory, and that enforce_eager can disable graph capture. It directs device selection to CUDA_VISIBLE_DEVICES, and warns that with tensor parallelism every process reads the whole checkpoint, making disk loading grow in proportion to the tensor-parallel size.

Start with an explicit command template:

CUDA_VISIBLE_DEVICES=YOUR_VALUE vllm serve YOUR_VALUE \
  --dtype auto \
  --gpu-memory-utilization 0.92 \
  --max-model-len YOUR_VALUE \
  --max-num-seqs YOUR_VALUE \
  --max-num-batched-tokens YOUR_VALUE \
  --tensor-parallel-size 1

Replace every YOUR_VALUE with the exact device selector, model identifier or path, and workload value. The command exposes the documented memory-fraction and tensor-parallel defaults so that you can change one workload setting without also changing the GPU allocation model.

On a single RTX 4090, keep --tensor-parallel-size 1. If you add GPUs later, account for the full-checkpoint reads described above before expecting model sharding to shorten loading. Set CUDA_VISIBLE_DEVICES before starting vLLM rather than selecting a device with torch functions, which the linked guide advises against. When memory is tight, lower context or sequence limits and record the workload reduction that accompanies each change.

Match batching to the workload

The vLLM optimization guide says V1 enables chunked prefill by default whenever possible and prioritizes decode requests. For max_num_batched_tokens, it gives 2048 as an example of a smaller value that can improve inter-token latency, says larger values can improve time to first token by processing more prefill tokens in a batch, and recommends a value above 8192 for optimal throughput, especially for smaller models on large GPUs. If chunked prefill is disabled, the guide requires max_num_batched_tokens to be greater than max_model_len.

Choose one primary goal before replacing the token-batch placeholder. Treat these as documented tuning directions rather than guaranteed results for your model: change one value, inspect the startup output and request behavior, and keep the selected setting in the run command.

When KV cache space runs out, the optimization guide says vLLM can preempt requests to free space and recompute them when enough space becomes available. In V1, the default preemption mode is RECOMPUTE rather than SWAP. Its stated remedy order is to increase gpu_memory_utilization first; if frequent preemption remains, decrease max_num_seqs or max_num_batched_tokens, which reduces concurrent work per batch and its KV cache demand.

Use --enforce-eager only as a deliberate startup choice. The guide says it skips compilation and CUDA-graph capture for the fastest possible startup, at the cost of steady-state decode performance. Remove it and recheck memory if continuous decoding matters more than startup speed.

Use quantization only when memory requires it

The vLLM quantization overview describes quantization as trading model precision for a smaller memory footprint and points to LLM Compressor, which supports FP8, INT8, INT4, and other formats. Its compatibility matrix marks AWQ as unsupported on Volta and AMD GPUs but supported on Turing, Ampere, Ada, and Hopper, while llm-compressor FP8 W8A8 is listed for Ada, Hopper, and AMD GPU columns. The table maps Ada to SM 8.9 and warns that compatibility can change.

Do not infer format support from the RTX 4090's 24 GB memory capacity alone. Match the exact model artifact against the current matrix, then validate task quality after quantization. The linked overview does not establish a universal conversion command, so do not add a quantization flag to the serving command unless your installed CLI documents it.

Verify the server and KV cache

The vLLM parallelism guide says the startup line GPU KV cache size reports the total number of tokens that can be stored in the GPU KV cache at once. Use it as a capacity signal, not as a performance result.

Verify the setup in this order:

  1. Launch the command and keep its console output available.
  2. Locate GPU KV cache size and record it beside your configured context, sequence, and token-batch limits.
  3. Confirm that the process remains active and that startup does not report a memory-allocation failure.
  4. Use the health or inference endpoint documented for your installed vLLM version. None of the settings above fixes a universal endpoint, so do not copy an unverified path.
  5. Exercise a request whose prompt and expected output fit within max_model-len, then repeat the request at your intended concurrency.

Treat the setup as working only when the process remains available, the startup report appears, and the documented request completes without unexpected preemption. If any condition fails, adjust the relevant workload limit before increasing concurrency.

Fix common failure patterns

  • Memory allocation fails during startup: Lower max_model_len or max_num_seqs, and reconsider CUDA-graph memory before adding parallelism.
  • Requests are repeatedly preempted and recomputed: Follow the documented remedy order; do not increase context, sequence, and token-batch limits together.
  • Startup is quick but steady-state decoding is weaker: Check whether --enforce-eager is enabled, then remove it if decoding performance is the priority.
  • The process sees the wrong GPU: Set CUDA_VISIBLE_DEVICES before launch instead of trying to select the device with torch functions.
  • A quantized checkpoint is rejected: Compare its format and target hardware with the current compatibility matrix rather than inferring support from memory capacity.
  • Loading becomes slower after adding GPUs: Remember that every tensor-parallel process reads the whole checkpoint, so disk-reading work increases with tensor-parallel size.

These steps are compiled from the official documentation linked in the text, and no commands or workloads were executed for this article; check them against the current version before use.

Sources