AI DeploySelf-hosted LLMGet it deployed

Tune Serverless GPU Cold Starts for LLM Inference

Reduce LLM cold starts when no ready container exists by tuning Modal and RunPod scaling, initialization, idle time, and verification.

For ai deployment, reduce cold starts by moving model-weight acquisition ahead of startup, defining a warm-capacity and idle-window policy, and measuring startup separately from steady inference. Treat this as a trade-off: capacity that remains ready can remove later starts but creates idle-resource charges, while a faster initialization path still has to preserve the generation behavior your workload needs.

Separate ai deployment delay from steady inference

Before changing settings, record expected peak concurrency, request duration, first-response target, generation behavior you must preserve, and acceptable idle spend. Classify each observed delay as queueing for capacity, initialization, time to first streamed output, or steady generation. Change only one control within a comparison; otherwise, you cannot attribute the result to capacity policy, initialization, or request mix.

Keep a decision note for every setting: which delay it targets, which behavior it may trade away, and how you will verify both. If the service must respond quickly after long idle gaps, prioritize warm capacity; if idle periods dominate and a slower first response is acceptable, allow capacity to scale down.

Tune Modal serverless warm capacity without hiding idle cost

Modal’s Cold start performance distinguishes time spent waiting for a warm container from initialization performed only on the first invocation, and recommends downloading model weights ahead of time. The same documentation identifies three warm-capacity controls: scaledown_window, min_containers, and buffer_containers. The documented maximum idle time is 60 seconds by default, while scaledown_window can range from 2 seconds to 20 minutes. A min_containers floor prevents the Function from scaling to zero, and buffer_containers adds capacity while the Function is active. Increasing scaledown_window reduces the chance of a later cold start, but resources used by the idle container, including a GPU reservation or residual memory occupancy, are billed.

Use min_containers only as a deliberate availability floor. Use buffer_containers when active traffic needs extra headroom, and use scaledown_window to make the latency-versus-idle-spend decision explicit. Do not copy values from another workload without matching its traffic pattern.

Tune the vLLM GPU initialization trade-off

Modal’s official OpenAI-compatible vLLM example caches Hugging Face weights in a Modal Volume so they are not downloaded on every server start. Its documented volume setup is:

hf_cache_vol = modal.Volume.from_name("huggingface-cache", create_if_missing=True)

The example also exposes a FAST_BOOT branch:

cmd += ["--enforce-eager" if FAST_BOOT else "--no-enforce-eager"]

When enabled, --enforce-eager disables Torch compilation and CUDA graph capture to favor faster boots at the cost of generation performance. Modal recommends FAST_BOOT=True for services that often scale from zero and False when several replicas are usually running. Apply the branch within the current official example rather than treating it as a standalone configuration.

After applying the example’s model, cache, and Function settings, its documented deployment command is:

modal deploy vllm_inference.py

Evaluate cache reuse and eager initialization separately. A faster boot is useful only if subsequent generation still meets the workload’s requirements.

Set RunPod serverless warm capacity and demand signals

RunPod’s Endpoint settings document that setting Active workers to 1 or more eliminates cold starts but incurs continuous charges, including while idle. The documented endpoint defaults are 0 Active workers, 3 maximum workers, a 5-second idle timeout, a 600-second execution timeout, a 24-hour job TTL, and FlashBoot enabled. FlashBoot retains worker state after spin-down and is described as most effective for endpoints with consistent traffic. Queue-delay scaling adds workers when requests exceed a threshold, which is 4 seconds by default. Request-count scaling uses Math.ceil((requestsInQueue + requestsInProgress) / scalerValue), is recommended for LLM workloads or frequent short requests, and uses a scaler value of 1 for maximum responsiveness. RunPod also suggests setting maximum workers about 20% above expected peak concurrency.

Choose an Active-worker floor for predictable baseline demand only when continuous idle charges fit the budget. For demand that arrives in bursts, compare buffer capacity with demand-triggered scaling. Fill the scaler value from your own queue behavior rather than assuming that the most aggressive setting is appropriate for every model.

Calculate the idle-versus-cold-start trade-off

Modal’s Billing guide says compute is charged when used or requested, without mandatory reservations or minimum usage-time increments. Workspaces are billed monthly, usage is charged after credits, and incremental charges can occur mid-cycle when specified thresholds are first exceeded. Subscription fees apply only to non-Starter plans. Use the current invoice or quote to fill this planning formula:

MODAL_CYCLE_ESTIMATE =
  YOUR_USAGE_CHARGE
  - YOUR_CREDITS
  + YOUR_INCREMENTAL_USAGE_CHARGES
  + YOUR_SUBSCRIPTION_FEES_IF_APPLICABLE

RunPod Serverless pricing bills from worker start until the worker fully stops, rounded up to the next second. Its billable worker time includes container initialization and model loading, request execution, and the idle timeout after a request, whose documented default is 5 seconds. Organize your own usage into those phases, then apply the provider’s quoted rate:

RUNPOD_BILLABLE_SECONDS =
  ROUNDUP(
    YOUR_START_SECONDS
    + YOUR_EXECUTION_SECONDS
    + YOUR_IDLE_SECONDS
  )

RUNPOD_WORKER_ESTIMATE =
  RUNPOD_BILLABLE_SECONDS × YOUR_PER_SECOND_RATE

If an invoice combines the phases, reconcile the calculation with the continuous worker charge instead of treating each phase as a separate invoice line. Compare cold-start and warm-capacity options over the same traffic window.

Verify LLM inference without measuring the wrong delay

The vLLM Benchmark CLI documentation documents an online vllm bench serve command against a running endpoint, supports ShareGPT for online and offline benchmarks, and recommends GuideLLM for production-server benchmarking. The documented command is:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

Use --request-rate to select a finite request rate or inf for maximum throughput. --max-concurrency limits outstanding requests and is unlimited when omitted. --burstiness shapes arrivals with a Gamma distribution, while --ramp-up-strategy accepts linear or exponential.

The report includes request throughput and output-token and total-token throughput. Time to first token, measured at the client, runs from sending the request to receiving its first streamed output. Inter-token latency records gaps between streamed outputs, while time per output token is calculated once per request and excludes the first token. With speculative decoding, one streamed output can contain several tokens, so these two latency measures can diverge. Repeating the benchmark against the same server may reuse prompts in the prefix cache and inflate throughput.

Because the documented command targets a running endpoint, do not label its result a cold-start measurement unless you separately control the provider’s scale-to-zero transition. Use the provider’s current documentation to trigger or confirm that transition; no undocumented trigger flag is assumed here. Record the client-side time to first token for the first request after scale-down, then compare later requests using the same model, inputs, and concurrency settings.

Catch common false wins

Before accepting a tuning result, check that:

  • Queue waiting and first-invocation initialization were measured separately.
  • Only one configuration changed within the comparison.
  • Faster startup did not introduce unacceptable steady-generation behavior.
  • Warm-capacity settings include their idle-cost implications.
  • First-after-idle and subsequent responses were recorded separately.
  • Repeated-server results account for possible prefix-cache reuse.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use. Nothing was run or measured for this article.

Sources