AI DeploySelf-hosted LLMGet it deployed

vLLM vs TGI: Self-Hosted LLM Serving Compared

Compare documented vLLM and TGI setup, APIs, batching, quantization, memory controls, and the fact that TGI is in maintenance mode.

vLLM and TGI can both expose /v1/chat/completions, but their documented choices are not equivalent: vLLM’s OpenAI-compatible server also covers completions, responses, embeddings, audio transcription and batch routes, while TGI’s consumption guide documents the chat route plus /generate and /generate_stream. TGI’s README status and feature overview says the project is in maintenance mode and lists continuous batching with Server-Sent Events; Hugging Face recommends vLLM, SGLang, llama.cpp and MLX going forward. For self hosted llm deployments, compare lifecycle, API contract, batching, memory controls and model format instead of turning these differences into an unsupported universal speed claim.

Prepare and start vLLM

The vLLM quickstart lists Linux and Python 3.10 to 3.13 as prerequisites. Its recommended installation uses a fresh Python 3.12 environment, while --torch-backend=auto lets uv inspect the installed CUDA driver and select the PyTorch index:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Start the quickstart’s example model with a single command:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The same quickstart places the default server at http://localhost:8000, provides --host and --port for changing the address, and states that one server process hosts one model at a time. If you expected to switch models inside that process, you need separate server instances.

For NVIDIA GPUs, the GPU installation guide requires CUDA compute capability 7.5 or higher and says the prebuilt package contains precompiled C++ and CUDA 12.9 binaries. The GPU installation source says Windows is not supported natively and directs Windows users to WSL. If you need a different CUDA version or an existing PyTorch installation, the NVIDIA CUDA installation notes direct you to build vLLM from source.

Start TGI

For NVIDIA GPUs, TGI’s container guidance requires the NVIDIA Container Toolkit and recommends NVIDIA drivers with CUDA 12.2 or higher. Its Docker example maps host port 8080 to container port 80, mounts storage at /data, supplies --model-id and reserves 1g of shared memory for NCCL. Set the HF_TOKEN environment variable in the server environment when accessing a gated or private model. The same documentation says CPU operation requires removing --gpus all and adding --disable-custom-kernels, but CPU is not the intended platform and performance may be poor.

Use your host volume path and model identifier in the documented container shape:

docker run --gpus all --shm-size 1g -p 8080:80 -v YOUR_VOLUME:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id YOUR_MODEL

Here, YOUR_VOLUME is the host storage location and YOUR_MODEL is the model passed to TGI. The image tag 3.3.5 is the tag shown by the linked README. Before composing a longer deployment command, inspect the container’s available deployment flags with the command from the TGI Quick Tour:

docker run ghcr.io/huggingface/text-generation-inference:3.3.5 --help

That help output covers areas such as sharding, quantization and generation parameters.

Verify the HTTP service

Start with vLLM’s documented model-list request:

curl http://localhost:8000/v1/models

The vLLM quickstart documents this as the way to list models. A response confirms that the HTTP server is reachable; a separate model request is still needed to confirm inference.

For operational checks, vLLM’s online-serving guide lists /version for version information, /load for server load metrics, /v1/models for available models and /health for health checking. Keep the model-discovery check separate from the real generation request so a running process cannot hide an incompatible model or payload.

The vLLM OpenAI-compatible server guide places text generation at /v1/completions and chat at /v1/chat/completions. The chat route requires a chat template and ignores the user parameter, while /v1/completions does not support suffix. A shared route name therefore does not make every client parameter interchangeable.

For TGI, the consumption guide specifies a POST request to /v1/chat/completions. Passing "stream": true requests a token stream, and TGI responses follow the OpenAI Chat Completion specification. Non-chat workloads can instead use /generate or /generate_stream.

Compare batching and backpressure

TGI’s continuous batching is paired with explicit admission and memory controls in the TGI launcher reference:

  • --max-concurrent-requests defaults to 128. A low limit refuses client requests instead of making them wait.
  • --max-total-tokens defines the per-request memory budget, covering the prompt and max_new_tokens.
  • --max-batch-total-tokens is inferred automatically when you do not provide it.

For vLLM, the optimization guide describes a different pressure point: when KV cache space runs out, vLLM can preempt requests and later recompute them. Its documented first remedy for frequent preemption is to increase gpu_memory_utilization, the share of GPU memory vLLM pre-allocates. These controls should be recorded in your runbook because they determine whether excess load is refused, delayed or recomputed; they are not benchmark results.

Which model and memory settings fit self hosted llm serving?

Start with the artifact you actually have. The TGI quantization guide separates pre-quantized weights—AWQ, GPTQ, Marlin and EXL2—from weights quantized on the fly with bitsandbytes, EETQ or fp8. Bitsandbytes needs no calibration dataset, but its inference is documented as slower than GPTQ or FP16 precision. Its documented 4-bit options are:

--quantize bitsandbytes-nf4
--quantize bitsandbytes-fp4

Do not map those TGI quantization names directly onto vLLM settings. Check the current vLLM documentation for the selected model artifact, then choose memory controls that match the deployment.

The vLLM memory guide documents several relevant controls. tensor_parallel_size can split a model across multiple GPUs, although every process reads the whole checkpoint, making disk loading time grow with the parallelism size. Select devices through CUDA_VISIBLE_DEVICES. Lowering max_model_len and max_num_seqs can reduce memory use, while CUDA graphs are enabled by default and consume additional GPU memory.

If startup memory or startup time matters, account for the trade-off described in the vLLM optimization guide: --enforce-eager skips compilation and CUDA-graph capture for the fastest startup, at the cost of steady-state decode performance.

Where deployments commonly go wrong

  • Port mapping: In the TGI Docker command above, host 8080 maps to container 80. The TGI launcher reference separately lists 3000 as the launcher’s own default. Check the published host mapping before changing application code.
  • Quantization format: The TGI quantization guide distinguishes pre-quantized artifacts from on-the-fly methods. The launcher reference also states that EXL2 does not support tensor parallelism when num_shard > 1.
  • Unsupported vLLM host: The GPU installation guide requires compute capability 7.5 or higher. For Windows, the installation source documents WSL rather than native Windows support.
  • Unexpected loading time: The vLLM memory guide warns that tensor-parallel processes each read the complete model before splitting it. Check checkpoint storage and device placement before assuming the server itself is faulty.

Before production, make an acceptance checklist from the exact package or image tag, model identifier, endpoint schema, streaming behavior, context and concurrency limits, startup check, model-discovery check, real inference request and health procedure.

These steps are compiled from the official documentation linked in the text; nothing was run or measured for them, and they should be checked against the current version before use.

Sources