AI DeploySelf-hosted LLMGet it deployed

Auto-Scale a Self-Hosted LLM Server with Traffic Data

Use vLLM request metrics and request history to set autoscaling signals, thresholds, and safe rollout checks for a self-hosted server.

For a self hosted llm server, use vLLM request metrics to identify saturation, publish the selected signal through a Kubernetes metrics API, and set scale-out and scale-in bands from your own request history. Apply the same GPU and model settings to every replica, and make readiness—not simple process existence—the gate for sending requests.

Prepare the GPU and serving boundary

Start with the resource boundary. After a vendor device plugin is installed, the Kubernetes GPU scheduling guide says the cluster exposes GPUs as a custom schedulable resource such as nvidia.com/gpu. Specify GPU quantities under limits; when the request is omitted, Kubernetes uses the limit as the request by default. If both fields are present, their values must match, and a GPU request without a limit is not allowed.

resources:
  limits:
    nvidia.com/gpu: YOUR_VALUE

Autoscaling cannot create schedulable GPU capacity. Verify that nodes can satisfy the GPU resource before increasing the desired replica count.

For online serving, build the vllm serve invocation from the options in the vLLM CLI reference. These are documented option fragments, not a complete command:

--gpu-memory-utilization 0.92
--max-model-len YOUR_VALUE
--max-num-seqs YOUR_VALUE
--max-num-batched-tokens YOUR_VALUE
--kv-cache-memory-bytes YOUR_VALUE

The documented default for --gpu-memory-utilization is 0.92, and the limit applies to the current vLLM instance. When --kv-cache-memory-bytes is set, it overrides the GPU memory utilization setting. Do not combine assumptions about both controls.

--max-model-len covers prompt and output context; without it, vLLM derives the length from the model configuration. A value of -1 or auto selects the largest model length that fits in GPU memory, but an explicit, validated value makes replicas easier to compare. Meanwhile, --max-num-seqs limits sequences processed in an iteration, while --max-num-batched-tokens limits tokens processed in an iteration.

Keep the selected settings consistent across replicas. Otherwise, a traffic increase could be handled by differently sized instances, making the metric history difficult to interpret.

Collect the traffic data that matters

The vLLM project's production metrics documentation exposes metrics at the following path on its OpenAI-compatible API server:

/metrics

The vLLM production metrics reference defines the main signals for a scaling policy:

Metric Documented meaning Role in an autoscaling policy
vllm:num_requests_running Gauge for requests in model execution batches Current demand
vllm:num_requests_waiting Gauge for requests waiting to be processed Queue pressure
vllm:kv_cache_usage_perc Gauge for KV-cache usage; 1 means 100% usage Memory saturation
vllm:time_to_first_token_seconds Histogram of time to first token Responsiveness guardrail
vllm:e2e_request_latency_seconds Histogram of end-to-end request latency User-impact guardrail
vllm:request_queue_time_seconds Histogram of time spent in the WAITING phase Queue-duration signal
vllm:num_preemptions Counter of engine preemptions Scheduling pressure
vllm:request_success Counter of successfully processed requests Outcome guardrail

Avoid turning every series into an opaque score without a clear reason. A clearer policy has a primary demand signal, a capacity signal, a latency guardrail, and an outcome check.

Set scaling bands for a self hosted llm from request history

Treat request history as the basis for thresholds rather than copying a universal constant. Record the selected metrics across representative quiet, normal, and peak traffic periods, keeping comparable request shapes together.

Use the history to answer several practical questions:

  • Choose a primary pressure signal. Waiting requests and queue time are useful when unfinished work is accumulating. KV-cache usage is useful when memory saturation is the main constraint.
  • Set a high band from the range your service can absorb. Cross it when the primary signal rises above the normal operating band or a latency guardrail deteriorates.
  • Set a low band from the range where work drains. Scale in only when the primary signal returns below that band, KV-cache pressure has room, and request outcomes remain healthy.
  • Keep the conditions interpretable. Record which metric caused each change instead of hiding the decision inside an unexplained composite score.

Treat cold starts separately from steady-state pressure. A newly added replica that is loading the model should be handled by probe timing, not treated automatically as evidence that the running fleet needs more capacity.

Review the policy whenever the model, vLLM serving flags, GPU allocation, or request mix changes. Historical bands remain useful only while the serving envelope remains comparable.

Connect the traffic signal to HPA

The Horizontal Pod Autoscaling guide says HPA can read metrics through these aggregated APIs:

metrics.k8s.io
custom.metrics.k8s.io
external.metrics.k8s.io

Custom metrics are supplied by adapter API servers from metrics solution vendors. Therefore, treat the vLLM /metrics endpoint as the collection point, not as an HPA metric identifier. Map the selected vLLM series through a documented adapter and use the metric name that adapter exports. The correct HPA object depends on that adapter and metric name, so check the current HPA schema and the adapter's current configuration examples.

The HPA control loop runs every 15 seconds by default. It also skips scaling when the current-to-target metric ratio is within 0.1 by default. Those are controller behavior, not workload thresholds: your traffic bands must still come from request history.

Use a scalable Kubernetes target rather than a DaemonSet, which HPA does not support. Confirm that the adapter, metric target, replica bounds, and GPU resources are all valid before interpreting a scale decision.

Make new replicas safe for traffic

Model loading and cache warm-up can take time, so the Kubernetes probe documentation separates several jobs:

  • A startup probe suits a container that takes a long time to become operational. If it fails, the kubelet kills the container and applies its restart policy.
  • A readiness probe controls whether a Pod receives traffic from matching Services. A failure marks the container not ready; it does not kill it.
  • A liveness probe determines when a container should be restarted, including when an application is deadlocked and unable to make progress.

Use the startup probe to protect normal initialization, the readiness probe to keep unready replicas out of rotation, and the liveness probe only for conditions that should trigger a restart. Added capacity should not receive requests merely because its container process exists.

Verify the loop and diagnose common failures

Check the complete chain

Verify each link in order:

  • Confirm that the selected vLLM series is present at the metrics endpoint.
  • Confirm that the adapter exposes the intended value and unit through a supported metrics API.
  • Replay representative request history until the high band is crossed.
  • Confirm that HPA changes the desired replica count and that new Pods can obtain their GPU resources.
  • Confirm that startup and readiness checks finish before matching Services send requests.
  • Watch waiting requests, queue time, KV-cache usage, latency, and request outcomes while traffic returns toward the low band.

Identify the failed link

If HPA cannot find the metric, check the adapter and exported metric name rather than repeatedly changing workload thresholds. If replicas remain pending, inspect GPU scheduling and the request-limit rules. If Pods run without receiving traffic, check readiness before treating the rollout as failed.

Repeated scale changes usually require a different policy: widen the gap between the high and low bands, choose a more relevant primary signal, or account for the controller interval. If queue pressure falls while latency worsens, add a user-impact guardrail. If successful-request counts fall, investigate the outcome instead of assuming that lower demand is healthy.

This article is compiled from the official documentation linked in the text; no commands or deployments were run for it. Check the current version before use.

Sources