Choose an LLM Deployment Architecture

Match model size, traffic, and operations to single-GPU, multi-GPU, multi-node, or cloud purchasing options using vLLM's documented controls.

Choose a single-GPU vLLM deployment when the model and its runtime memory fit on one supported device; use tensor or pipeline parallelism when the model must span GPUs, and consider multi-node deployment when it must span machines. For ai deployment, separate model fit, GPU placement, request scheduling, and purchase commitment instead of treating “cloud” or “bare metal” as a complete architecture. The vLLM guidance below provides a concrete serving baseline; Kubernetes and serverless choices still require current platform-specific documentation.

Turn deployment constraints into a topology

A sound ai deployment starts with independent questions:

Model fit: What context length and concurrency must the model support, and how much memory will they require?
Placement: Does the model fit on one GPU, need several GPUs in one node, or span multiple nodes?
Operations: Which layer will own process startup, restarts, endpoint exposure, and request routing?
Purchase: Will you use available capacity, or accept a usage or configuration commitment?

An inference-engine setting does not settle the hosting architecture. Before choosing Kubernetes or a serverless platform, consult its current documentation for process lifecycle, network exposure, scaling limits, and billing.

Use a worksheet rather than treating a single GPU specification as proof that a model will fit. This is an editorial planning method, not an official vLLM sizing formula:

GPU_MEMORY_PLAN =
  MODEL_WEIGHT_MEMORY
  + YOUR_KV_CACHE_RESERVE
  + YOUR_RUNTIME_AND_GRAPH_OVERHEAD

HOST_MEMORY_PLAN =
  YOUR_BASE_HOST_MEMORY
  + MULTIMODAL_PROCESSOR_CACHE

Fill the placeholders from the model project’s documentation, the hardware specifications, and the workload you actually need to serve. Keep enough KV-cache capacity for the intended context and concurrency rather than optimizing every setting in isolation.

Prepare a clean vLLM host

The vLLM quickstart requires Linux and Python 3.10 to 3.13. It recommends a fresh Python 3.12 environment created with uv; --torch-backend=auto inspects the installed CUDA driver when selecting the PyTorch index.

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

For NVIDIA GPUs, the GPU installation guide requires compute capability 7.5 or higher. Its prebuilt package contains pre-compiled C++ and CUDA 12.9 binaries. If you need a different CUDA version or want to retain an existing PyTorch installation, the guide directs you to build vLLM from source; that build requires GCC/G++ 11.3 or newer.

A fresh environment is an architectural boundary as well as an installation convenience. It prevents an existing CUDA or PyTorch configuration from silently determining the serving stack.

Serve one model through one server process

Start the server with the documented model command:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

If your design requires several models, plan a separate server process for each one rather than assuming that a single process will switch between them. Decide separately how those processes will be exposed, routed, and restarted.

Keep the initial endpoint local while checking the deployment. If application users need remote access, add the access controls, network policy, and operating procedures required by your environment before binding a public interface.

Choose single-node GPU parallelism

For a multi-GPU node, the vLLM parallelism guide distinguishes two placements. Tensor parallelism shards model parameters across GPUs within each model layer and is the common single-node strategy. Pipeline parallelism distributes model layers across GPUs, with each GPU processing its part in sequence. Where the node’s GPUs lack NVLINK, the guide recommends pipeline parallelism over tensor parallelism for higher throughput and lower communication overhead.

Before launch, the vLLM memory guidance says to control selected devices with CUDA_VISIBLE_DEVICES rather than by calling PyTorch device functions before vLLM starts. With tensor parallelism, every process reads the whole model checkpoint and divides it into chunks, so disk-reading time grows in proportion to the tensor-parallel size.

That loading behavior belongs in the architecture decision. A multi-GPU arrangement can divide model parameters, but it does not divide the initial checkpoint read performed by each tensor-parallel process.

Expand beyond one node only when required

For multi-node inference, the same parallelism guide gives a concrete rule: set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes. It identifies Ray as the default distributed runtime for multi-node inference and native Python multiprocessing for single-node inference.

The same guide identifies the startup line GPU KV cache size as the total number of tokens that fit in the GPU KV cache at once. Use that line as an architecture check: a server can start while still having less cache capacity than the intended context and concurrency require.

Size memory and request scheduling deliberately

The memory guidance says memory usage can be reduced by limiting max_model_len and max_num_seqs. CUDA graphs are enabled by default and consume additional GPU memory; enforce_eager disables graph capture completely. Use those controls deliberately, because a lower context or batch limit also changes what the service can accept.

The vLLM optimization guide explains that when KV-cache space runs out, vLLM can preempt requests and recompute them later. In V1, the default preemption mode is RECOMPUTE. Its first remedies are to increase gpu_memory_utilization, which controls the share of GPU memory pre-allocated for cache, or to decrease max_num_seqs or max_num_batched_tokens so a batch needs less KV-cache space.

Scheduling settings also involve a workload trade-off. The optimization guidance says smaller max_num_batched_tokens values can improve inter-token latency, while higher values can improve time to first token by processing more prefill tokens together. If chunked prefill is disabled, max_num_batched_tokens must be greater than max_model_len. Record these choices in the architecture record instead of changing them blindly during an incident.

Separate architecture from purchase commitment

The AWS EC2 purchasing-options page says On-Demand Instances are billed by the second. It describes Savings Plans as commitments to consistent usage measured in USD per hour for a term of one or three years. Reserved Instances commit to a specific instance configuration, including instance type and Region, for one or three years, while Capacity Blocks can reserve a cluster of GPU instances.

Use the provider’s current page and your own quote to complete this checklist:

Confirm whether the model and operating pattern justify a one-year or three-year commitment.
Record the exact instance configuration and Region for any reserved purchase.
Identify whether Capacity Blocks apply to the required GPU-cluster arrangement.
Copy every charge shown on the quote or bill rather than estimating from the inference rate alone.

TOTAL_PROVIDER_COST =
  ON_DEMAND_CHARGES
  + SAVINGS_PLAN_CHARGES
  + RESERVED_INSTANCE_CHARGES
  + CAPACITY_BLOCK_CHARGES
  + OTHER_ITEMS_ON_YOUR_QUOTE_OR_BILL

Repeat the same method for another cloud, Kubernetes, or serverless provider using its current documentation. A shorter commitment, a different instance configuration, and a different traffic profile belong in separate scenarios; do not combine them into an unsupported total.

Verify the path and diagnose common failures

The vLLM quickstart documents http://localhost:8000 as the default address, with --host and --port controlling the listener. One server process hosts one model at a time, and the documented model-list request is:

curl http://localhost:8000/v1/models

A successful listing confirms that the HTTP route is responding, but it does not replace a real generation request. Use your application’s documented client and request format to verify generation, then compare the observed behavior with the context, concurrency, and memory plan.

When troubleshooting, work through the architecture layers in order:

If the wrong devices are visible, revisit CUDA_VISIBLE_DEVICES.
If requests are repeatedly preempted, revisit KV-cache allocation, max_num_seqs, and max_num_batched_tokens.
If multi-GPU loading is slower than expected, account for each tensor-parallel process reading the whole checkpoint.
If context and batch settings conflict, check the documented relationship required when chunked prefill is disabled.
If the server binds incorrectly, verify --host and --port before changing GPU settings.

These steps are compiled from the official documentation linked in the text, were not run or benchmarked for this article, and should be checked against the current version before use.