vLLM vs TensorRT-LLM for NVIDIA LLM Serving

A practical comparison of vLLM and TensorRT-LLM setup, memory controls, supported NVIDIA hardware, APIs, and deployment checks.

For a self hosted llm on NVIDIA GPUs, vLLM provides a direct vllm serve command through its serve CLI reference, while TensorRT LLM can serve models through its Triton TensorRT-LLM backend. Choose by matching model loading, your GPU against the listed supported hardware, the required API, and controllable memory settings; these pages do not establish a common benchmark or universal speed winner.

A self hosted llm decision for NVIDIA serving

Start with the serving boundary, because a direct server and a Triton-managed deployment expose different control surfaces.

Standalone vLLM: Online engine arguments are supplied directly to vllm serve, according to the vLLM engine-argument documentation. This is the shorter path when you want to control the server process without adding a model-repository layer.
Direct TensorRT LLM: The TensorRT LLM quick start identifies trtllm-serve as the command for starting an OpenAI-compatible server for a model.
Triton-managed serving: Triton organizes models through a model repository and provides HTTP/REST and GRPC interfaces based on KServe, as described in the Triton Inference Server README.

A direct process has fewer deployment layers. Triton adds a repository layout, backend selection, model configuration, and protocol boundary, but can be useful when those layers are part of your serving design.

What to prepare

The vLLM quickstart lists Linux and Python 3.10 to 3.13 as prerequisites. Its GPU installation guide requires NVIDIA GPU compute capability 7.5 or higher and states that the pre-built package contains pre-compiled C++ and CUDA 12.9 binaries. Use a fresh environment when possible: the CUDA-specific installation guidance says a different CUDA version or an existing PyTorch installation requires building vLLM from source. For Blackwell B200 and GB200 GPUs, the GPU guide also requires at least CUDA 12.8.

TensorRT LLM is described by its official overview as an NVIDIA open-source library for optimizing LLM inference on NVIDIA GPUs. Its hardware page lists Blackwell B200, GB200, B300, GB300, and DGX Spark; Hopper H100, H200, and GH200; Ada Lovelace L20 and L40/L40S; and Ampere A100. The overview separately states that FP8 quantization is supported on H100 and later GPUs.

If you choose Triton, be ready to manage its repository and container structure. The Triton README recommends Docker images and lists support for TensorRT, PyTorch, ONNX, OpenVINO, Python, and RAPIDS FIL across NVIDIA GPUs, x86 and ARM CPUs, or AWS Inferentia.

Install and check vLLM

The vLLM quickstart documents this environment and server sequence. With --torch-backend=auto, uv inspects the installed CUDA driver version when choosing the PyTorch index.

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Start the documented model:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The server defaults to http://localhost:8000; use the documented --host and --port flags if the listener must differ. The quickstart also states that one server process hosts one model at a time.

Check the model-listing endpoint:

curl http://localhost:8000/v1/models

A response confirms that the HTTP server and model-registration path are reachable. It does not replace sending a valid inference request and checking that response.

Control vLLM memory and parallelism

The vLLM serve CLI reference defines --gpu-memory-utilization as the per-instance fraction of GPU memory assigned to the model executor. Its documented range is 0 to 1, with a default of 0.92. Other useful controls include:

--max-model-len, which covers prompt plus output and otherwise comes from the model configuration. A value of -1 or auto asks vLLM to select the largest length that fits in GPU memory.
--max-num-seqs, which caps sequences processed in one iteration.
--max-num-batched-tokens, which caps tokens processed in one iteration.
--tensor-parallel-size or -tp, and --pipeline-parallel-size or -pp, both documented with a default of 1.
--kv-cache-memory-bytes, which sets KV-cache memory per GPU and overrides --gpu-memory-utilization when supplied.

A practical configuration can expose only the values you intend to fill:

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --gpu-memory-utilization YOUR_VALUE \
  --max-model-len YOUR_VALUE \
  --max-num-seqs YOUR_VALUE

The vLLM memory guide identifies lower context and sequence limits as ways to reduce memory use. It also directs device selection through CUDA_VISIBLE_DEVICES rather than calling PyTorch device functions before vLLM starts. With tensor parallelism, every process reads the complete checkpoint before splitting it, so disk-loading time grows with the tensor-parallel size.

For a multi-node arrangement, the parallelism and scaling guide defines tensor_parallel_size as GPUs per node and pipeline_parallel_size as the number of nodes. If startup matters more than steady-state decoding, --enforce-eager disables compilation and CUDA-graph capture; the optimization guide describes this as the fastest-startup option at the cost of steady-state decode performance.

Set up the TensorRT LLM path

Current TensorRT LLM development has moved away from the legacy engine workflow. Its migration guide states that the TensorRT engine backend has been removed, PyTorch is now the sole execution backend, and trtllm-build, trtllm-refit, and trtllm-prune are no longer used. HuggingFace checkpoints load directly without an engine-build step.

The installation guide documents running the pre-built NGC release container with host IPC, GPU access, memory-locking settings, and port 8000 exposed:

docker run --rm -it \
  --ipc host \
  --gpus all \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -p 8000:8000 \
  nvcr.io/nvidia/tensorrt-llm/release:YOUR_VALUE

Replace YOUR_VALUE with the release tag specified by the installation material you are using. Inside the prepared environment, the TensorRT LLM quick start identifies trtllm-serve as the OpenAI-compatible server entry point and documents /v1/chat/completions on port 8000.

To verify that path, send the POST request defined by the current quick start and confirm that the endpoint returns an OpenAI-compatible response. The cited quick-start entry establishes the endpoint and request header but does not provide a complete JSON request body here, so use the current page rather than inventing model or payload fields. It also does not identify a separate health-check endpoint.

Put either engine behind Triton

For the vLLM integration, the Triton vLLM backend README describes a Python-based backend. Requests are passed to vLLM's AsyncEngine when received, while the engine handles inflight batching and paged attention. The backend’s model.json file supplies key-value arguments used to initialize AsyncLLMEngine.

Triton’s own dynamic-batching documentation says the server can combine incoming requests into a dynamically created batch for stateless models. An empty dynamic-batching block uses the largest batches possible up to the maximum batch size without delaying batch formation. This documented fragment shows how a limited queue delay can be configured:

dynamic_batching {
  max_queue_delay_microseconds: 100
}

The value shown is an example, not a capacity recommendation.

For an OpenAI-compatible frontend, NVIDIA documents this sample command:

python3 openai_frontend/main.py \
  --model-repository tests/vllm_models \
  --tokenizer meta-llama/Meta-Llama-3.1-8B-Instruct

The OpenAI-compatible frontend documentation places /v1/chat/completions on port 9000. Treat tests/vllm_models and the named tokenizer as sample paths from the project, not as a production repository layout.

Where it usually goes wrong

Repository structure: Triton requires each model directory to contain at least one numeric version subdirectory, according to the model-repository guide. Model configuration normally belongs in config.pbtxt as a ModelConfig protobuf, as explained in the model-configuration guide.
Multi-GPU and memory assumptions: The Triton vLLM backend requires kind: KIND_MODEL for multi-GPU models. Its README also says vLLM may consume up to 90% of GPU memory under default settings and that the sample model sets gpu_memory_utilization to 50%; do not copy the sample value without checking how your direct server is configured.
Legacy TensorRT commands: Do not plan around trtllm-build, trtllm-refit, or trtllm-prune. The migration guide marks them as removed and directs current workflows toward direct HuggingFace checkpoint loading.
Frontend feature assumptions: The OpenAI-compatible frontend enables tool calling with --tool-call-parser; the documented parsers are llama3 and mistral. Its embedding endpoints currently work through the vLLM backend, so an embedding requirement can change the backend decision.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.