AI DeploySelf-hosted LLMGet it deployed

Compare vLLM, TGI, Triton, and Ray Serve for Self-Hosting

Compare vLLM, TGI, Triton, and Ray Serve by role and setup, including Triton's ability to host vLLM behind its model-serving layer.

For a self hosted llm, the practical split is vLLM as a direct model server (vLLM Quickstart), TGI as a separate inference server whose project README places it in maintenance mode, and Triton as a broader model-serving layer (Triton Inference Server README). Ray Serve is not covered by the cited project pages, so this comparison does not invent an installation command, API, or runtime role for it.

Decide which self hosted llm layer you need

Start with deployment responsibility rather than a feature count. A direct model server, a text-generation server, and a model-repository or protocol layer expose different controls; treating them as interchangeable hides where model loading, batching, and request compatibility are configured.

For any additional serving layer, require current documentation for its installation, model-loading process, request protocol, batching behavior, and health checks before treating it as equivalent to the documented options below.

Prepare the self-hosting host and model access

Before starting, inventory the operating system, accelerator, driver, container runtime, model source, credentials, and persistent cache. The vLLM GPU installation guide says Windows is not supported natively and points Windows users to WSL with a compatible Linux distribution. For NVIDIA GPUs, it requires compute capability 7.5 or higher.

That guide also states that the prebuilt package contains C++ and CUDA 12.9 binaries. A different CUDA version or an existing PyTorch installation requires building vLLM from source, for which GCC/G++ 11.3 or newer is required. Treat these as version-sensitive installation gates rather than optional guidance.

Run vLLM directly for one served model

The vLLM Quickstart lists Linux and Python 3.10 to 3.13 as the standard prerequisites. With uv available, its recommended fresh Python 3.12 environment is:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

The --torch-backend=auto option lets uv inspect the installed CUDA driver and select the appropriate PyTorch index. Start the documented model server with:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The server listens at http://localhost:8000 by default; --host and --port change the address. One server process currently hosts one model at a time. API-key checking can be enabled with --api-key or VLLM_API_KEY, and multiple keys can follow --api-key for key rotation.

Check the running model through the documented models endpoint:

curl http://localhost:8000/v1/models

An OpenAI Python client can instead be directed to the server by setting its base_url to http://localhost:8000/v1.

Run TGI as a standalone inference server

The TGI README describes TGI as a Rust, Python, and gRPC server. It also says the project is in maintenance mode and accepts only minor bug fixes, documentation improvements, and lightweight maintenance work.

Preserve the README’s documented container structure and replace both YOUR_VALUE placeholders with your data-volume path and model ID:

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v YOUR_VALUE:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id YOUR_VALUE

The shown image tag is 3.3.5; confirm the current tag before reuse. The port mapping exposes container port 80 on host port 8080. The --shm-size 1g setting gives the container shared memory for NCCL sharing.

NVIDIA container use requires the NVIDIA Container Toolkit, and the README recommends drivers with CUDA 12.2 or higher. For gated or private models, set HF_TOKEN. TGI documents token streaming through Server-Sent Events and continuous batching of incoming requests.

The CPU path is obtained by removing --gpus all and adding --disable-custom-kernels, but the README explicitly says CPU is not the intended platform for the project.

Use Triton as the model-repository and protocol layer

The Triton Inference Server README describes support for frameworks including TensorRT, PyTorch, ONNX, and OpenVINO. It lists NVIDIA GPUs, x86 and ARM CPUs, and AWS Inferentia as supported hardware, recommends Docker images, and bases its HTTP/REST and gRPC inference protocols on KServe.

The documented server launch syntax is:

tritonserver --model-repository=/models \
  --model-control-mode explicit \
  --load-model densenet_onnx

This command demonstrates the repository, explicit model-control, and model-loading flags; densenet_onnx is not a complete LLM deployment recipe. Use the current container instructions to mount the intended model repository at /models.

The Triton vLLM backend README describes a Python backend that sends requests directly to vLLM’s AsyncEngine, where inflight batching and paged attention are handled. Its model.json file supplies key-value arguments when initializing AsyncLLMEngine. Multi-GPU models require the instance-group kind KIND_MODEL in config.pbtxt.

Memory settings also need attention: the backend documentation says vLLM can consume up to 90% of GPU memory under its defaults, while the sample model sets gpu_memory_utilization to 50%. These are documented defaults and sample settings, not capacity guarantees.

Verify Ray Serve before treating it as equivalent

The cited pages do not establish Ray Serve’s installation command, supported runtime, deployment API, model-loading process, request protocol, batching behavior, or health-check endpoint. This article therefore makes no feature-parity claim about it.

Before including Ray Serve in the same decision, record the following from its current official documentation:

  • Installation command and supported runtime versions.
  • Deployment definition and model-loading path.
  • Request adapter, batching, and streaming controls.
  • Health checks and GPU or resource configuration.
  • Health endpoint and example request.

If a feature is not stated explicitly, leave it unresolved rather than inferring it from the other projects.

Verify each documented interface

Use each service’s documented interface rather than assuming that similar model-serving tools expose the same URL or request format.

For TGI, the consuming guide identifies these routes:

GET /docs
POST /v1/chat/completions
/generate
/generate_stream

The /docs route serves the REST API’s OpenAPI documentation. A chat request uses POST /v1/chat/completions, and setting "stream": true requests a token stream. Responses from that route follow the OpenAI Chat Completion specification; /generate and /generate_stream serve non-chat use cases.

Triton’s cited launch documentation provides an explicit --load-model selection but no fixed health URL. Use the KServe-based protocol and the configuration for the deployed model instead of assuming the vLLM /v1/models route exists in Triton.

Common failure points

vLLM: build and chat-template errors

If the prebuilt installation does not match your CUDA version or PyTorch setup, the documented route is a source build. The NVIDIA CUDA installation notes also state that installing nightly wheels with pip is unsupported and that uv is required.

Chat requests fail when the tokenizer has no chat template. The online serving guide says --chat-template accepts either a file path or the template string, which gives you an explicit way to supply one.

TGI: container assumptions

Recheck the image tag, NVIDIA driver and container access, shared-memory setting, model ID, and HF_TOKEN before changing server behavior. A CPU launch is possible through the documented flags, but it is not the project’s intended platform.

Triton: repository structure and batching

The model repository guide says --model-repository can be supplied more than once, but every model directory needs at least one numeric version subdirectory. Model settings are normally supplied through config.pbtxt as a ModelConfig protobuf, according to the model configuration guide.

Dynamic batching is intended for stateless models. The batchers guide provides this configuration form:

dynamic_batching {
  max_queue_delay_microseconds: 100
}

The key lets requests wait in the scheduler so other requests can join the batch. Triton recommends not setting preferred_batch_size for most models.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use. Nothing was run or measured for this article.

Sources