AI DeploySelf-hosted LLMGet it deployed

Reduce Self-Hosted LLM Inference Latency End to End

Trace self-hosted LLM latency through vLLM scheduling, KV-cache pressure, streaming gateways, and end-to-end benchmarks with documented checks.

You can reduce inference latency in a self hosted llm stack by controlling KV-cache pressure, aligning scheduler settings with the latency target, and verifying that the gateway does not buffer streamed tokens. Treat time to first token and the interval between later output tokens as separate objectives, benchmark the same client-to-vLLM path used by the application, and change only the setting tied to the symptom you are trying to solve.

Where self hosted llm latency is created

MLPerf defines request latency from the moment LoadGen passes a query to the system under test until a reply arrives. For LLM benchmarks, it separates time to first token (TTFT) from time per output token (TPOT), the average interval between generated tokens; the MLPerf Inference Rules define these measures.

At the client, record when the request was sent, when the first token arrived, the pattern of later token arrivals, and when the final response completed. Use the same timestamps for direct and proxied runs so that gateway delay does not disappear inside an engine-only result.

For rented GPUs, region placement remains provider- and workload-specific. Use the provider’s current documentation to confirm region availability, the client-to-instance network path, and region-specific billing. Record the provider, region, and client location in the run notes so a network change is not mistaken for a model improvement.

Prepare the hardware and model

Start with topology rather than flags. vLLM’s Parallelism and Scaling guide describes tensor parallelism as sharding model parameters across GPUs within each model layer and calls it a common single-node strategy. Pipeline parallelism distributes different layers across GPUs and processes them in sequence; without NVLINK, the guide recommends pipeline parallelism for higher throughput and lower communication overhead.

The Conserving Memory guide notes that every tensor-parallel process reads the whole checkpoint and splits it into chunks, making disk-reading time grow in proportion to tensor-parallel size. It also directs you to select devices through this environment variable rather than call torch device functions before vLLM starts:

CUDA_VISIBLE_DEVICES

Record the selected device IDs, GPU count, and interconnect in the run configuration. This prevents a later CPU or storage change from being mistaken for the result of an engine setting.

Launch a documented vLLM baseline

The OpenAI-Compatible Server documentation provides this launch example:

vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key YOUR_VALUE

Treat it as a command-syntax example, not a recommendation that the named model fits your hardware. Keep the exact model identifier and launch command in the deployment record, and protect the API key outside the command configuration rather than committing a real value to a run file.

Reduce model-side latency

Start with the memory budget. The Conserving Memory guide says that lowering max_model_len and max_num_seqs can reduce memory usage. It also says CUDA graphs are enabled by default and consume extra GPU memory, while enforce_eager turns graph capture off completely.

Review these exact engine controls before changing the workload:

max_model_len
max_num_seqs
max_num_batched_tokens
gpu_memory_utilization

Choose values from the context, concurrency, and memory available in your deployment rather than copying a profile made for another model. Treat enforce_eager as a trade-off, not an unconditional latency improvement.

The Optimization and Tuning guide explains why max_num_batched_tokens needs a workload-specific value. In V1, chunked prefill is enabled by default whenever possible, and the scheduler prioritizes decode requests. A smaller value, such as 2048, can improve inter-token latency because fewer prefills slow decodes; a higher value can improve TTFT by processing more prefill tokens in a batch. The guide recommends values above 8192 for optimal throughput, especially for smaller models on large GPUs. If chunked prefill is disabled, max_num_batched_tokens must be greater than max_model_len.

Cache exhaustion requires a different response. vLLM can preempt requests when KV-cache space runs out and recompute them when sufficient space becomes available. The same optimization guidance lists increasing gpu_memory_utilization, which controls the GPU-cache share preallocated by vLLM, or decreasing max_num_seqs or max_num_batched_tokens to reduce concurrent batch demand and KV-cache use. Apply those remedies to diagnosed cache pressure rather than changing them blindly.

Verify the model and API path

Read the GPU KV cache size startup line. The Parallelism and Scaling guide says it reports the total number of tokens that can be stored in the GPU KV cache at once. Compare that capacity with the intended context and concurrency; treat it as a capacity diagnostic rather than a latency result.

The documented generation routes are:

/v1/completions
/v1/chat/completions
/v1/responses

The OpenAI-Compatible Server documentation says /v1/completions applies to text-generation models and does not support suffix. The Chat Completions route requires a chat template and ignores the user parameter. Exercise the same route and request shape used by the application, then record the client-observed first-token delay and subsequent token pattern.

Remove gateway buffering

The NGINX proxy module documentation says proxy_buffering is enabled by default, causing the proxy to read the upstream response as quickly as possible into buffers. For token streaming or SSE delivery, use the documented synchronous path:

proxy_buffering off;

With buffering disabled, NGINX passes the response to the client synchronously as it is received. The upstream application can instead control buffering per response with this header:

X-Accel-Buffering: no

The same NGINX module documentation gives proxy_read_timeout a default of 60s. It applies between successive read operations, not to the entire response, so a long response can continue while reads remain active. If the client receives output in bursts, recheck streaming behavior before treating total generation duration as a timeout problem.

Benchmark the end-to-end path

An end-to-end run includes the client, network, proxy, scheduler, model execution, and response stream. A direct-to-vLLM run isolates the engine, but it does not represent the complete path seen by the application.

The MLPerf Inference Rules require every submission to use LoadGen and define the system under test as a defined set of hardware and software resources. They specify Poisson query arrivals for the Server scenario, all Offline samples sent at the start, and the next Single query sent after the previous request completes. Offline has no latency constraint; Single uses a 90th-percentile early-stopping latency estimate; Server reports the maximum Poisson throughput supported.

Choose the scenario that matches the traffic you need to understand rather than treating batch and interactive results as interchangeable. Use the command documented by the benchmark implementation you select; the cited rules define the scenarios but not an invocation for your particular deployment.

For every controlled run, preserve:

  • The model identifier, revision, vLLM version, and complete configuration.
  • GPU model and count, interconnect, and CPU topology.
  • Provider, region, and client location.
  • Prompt and output shape, concurrency, and streaming behavior.
  • Warm-up policy, run boundaries, TTFT, and TPOT results.

Keep a direct engine run beside the proxied run during operational diagnosis, but do not present the direct result as end-to-end latency.

Common failure patterns

  • Collapsing latency measures: Keep TTFT, TPOT, request completion, and client timestamps distinct.
  • Changing engine controls together: Preserve the exact configuration and tie each run to a deliberate change.
  • Treating startup as success: Exercise the application’s API route and streaming path before benchmarking.
  • Comparing unlike runs: Keep the provider, region, request shape, concurrency, and benchmark scenario fixed.
  • Chasing engine tuning when delay appears only through the proxy: Recheck the gateway settings before changing model configuration.

Classify the symptom as occurring before the first token, between output tokens, or only through the proxy. Then change a deliberate setting and rerun the same path so the result remains attributable.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources