AI DeploySelf-hosted LLMGet it deployed

Serverless GPU Cold Starts for LLM APIs

See how a request waits when no worker is ready, which controls reduce startup, and how to compare latency, billing, and throughput.

Serverless GPU cold starts are the delay created when an API request finds no ready inference worker and must wait for startup; for llm hosting, the main levers are model caching, initialization placement, warm-capacity settings, and the provider’s billing boundary. A useful evaluation separates the zero-worker request from the next warm request, then checks whether startup and idle time are billable instead of treating GPU type as the whole answer. Keep the model, request shape, dataset, and concurrency unchanged while changing only the scaling or initialization control under test.

Prepare an llm hosting test that separates cold from warm

Before changing settings, define the cold state precisely. Do not infer it from client silence: verify that the endpoint has no running worker, then send a fixed request and record whether the platform starts capacity, queues the request, or rejects it. If the request is accepted, capture send, first-response, and completion timestamps. Repeat the request while capacity is warm and retain those events as a separate observation.

Use a record such as:

provider = YOUR_VALUE
model_and_revision = YOUR_VALUE
request_or_dataset = YOUR_VALUE
zero_capacity_evidence = YOUR_VALUE
cold_request_elapsed = YOUR_VALUE
next_warm_request_elapsed = YOUR_VALUE
billable_start_time = YOUR_VALUE
billable_idle_time = YOUR_VALUE

Also save the active-worker, maximum-worker, timeout, initialization, and snapshot settings in effect. This prevents a capacity change, a model-loading change, and a traffic change from being mistaken for the same result.

On Modal, cache weights before tuning capacity

Modal’s official vLLM example runs vLLM in OpenAI-compatible mode and caches Hugging Face weights in a Modal Volume so they are not downloaded on every server start. It creates the named Volume on demand and deploys the file with modal deploy; the page also notes that, unlike a Modal Function, a Modal Server rejects requests with 503 Service Unavailable when no containers are active.

hf_cache_vol = modal.Volume.from_name("huggingface-cache", create_if_missing=True)

modal deploy vllm_inference.py

The example’s FAST_BOOT switch adds --enforce-eager when true. That disables Torch compilation and CUDA graph capture to favor faster boots at the cost of generation performance. Modal recommends true for services that often scale from zero and false when several replicas are usually running.

cmd += ["--enforce-eager" if FAST_BOOT else "--no-enforce-eager"]

For verification, start from the endpoint’s documented zero-container state and use your existing OpenAI-compatible client. Keep rejected requests separate from accepted inference requests, then compare the accepted cold request with the next request made while capacity remains warm.

Separate Modal queue time from first-use work

Modal’s cold-start performance guide separates time spent waiting for a warm container from extra work performed only on the first invocation. It says containers boot in about one second, but global-scope code and all modal.enter methods must finish before inputs are routed; moving work into enter shifts latency into warm-up rather than removing it.

The same guide identifies scaledown_window, min_containers, and buffer_containers as the warm-capacity controls. The default maximum idle time is 60 seconds, and scaledown_window can range from 2 seconds to 20 minutes. A longer window reduces the chance of another cold start but leaves idle resources billable, min_containers sets a floor, and buffer_containers adds capacity while the Function is active.

Change these controls independently. For example, keep weight caching fixed while testing scaledown_window, then keep capacity settings fixed while changing initialization. Naming the changed variable makes a slow or fast run easier to diagnose.

On RunPod, budget the worker from start to stop

RunPod’s Serverless pricing guide bills from worker start until the worker fully stops, rounded up to the next second. Its billable interval has three parts: start time for container initialization and model loading into GPU memory, execution time, and the idle-timeout period, which defaults to 5 seconds.

Use the provider’s bill or quote to fill this duration calculation:

billable_worker_duration = round_up(start_time + execution_time + idle_timeout)

Client-visible response time does not show the whole billing interval. Preserve the worker start and stop events as well, then classify the intervening time as startup, execution, or idle according to the pricing page.

Read RunPod worker controls as a latency-and-cost decision

RunPod’s endpoint settings say that a floor of 1 or more active workers eliminates cold starts, while active workers are charged continuously, including while idle. FlashBoot is enabled by default; it retains worker state after spin-down and is most effective for endpoints with consistent traffic where workers frequently cycle between active and idle states.

The same settings page lists defaults of 0 active workers and 3 maximum workers. It suggests setting maximum workers about 20% above expected peak concurrency and documents queue-delay scaling with a default threshold of 4 seconds. Its request-count scaling method calculates:

Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)

For each trial, verify that no worker was running before dispatching the cold request. Make the warm request immediately after capacity becomes ready, and save the active-worker count, maximum-worker count, FlashBoot state, and scaling method beside the timing result. Change only the setting being evaluated.

Classify Replicate models before comparing cold-start cost

Replicate’s billing guide says a model with no demand scales down to its configured minimum instance count, which is 0 by default. After a gap in requests, the instance remains idle for a few minutes rather than shutting down immediately. Public models use a hardware pool and queue shared with other customers by default.

The same billing policy says public models are charged only for active processing time, with setup and idle time free. Private models and deployments are charged for setup, idle, and active time; most private models use dedicated hardware without a shared queue. Fast-booting fine-tunes are the documented exception: they are charged only while actively processing, whether public or private.

Represent the billing categories separately:

public_model = active_processing_time
fast_booting_fine_tune = active_processing_time
private_model_or_deployment = setup_time + idle_time + active_processing_time

Check the model page’s Run time and cost section before estimating a charge. Do not carry an active-time assumption from a public model into a private model or deployment without confirming its billing category.

Measure first-request latency separately from vLLM throughput

The vLLM Benchmark CLI documentation supports ShareGPT for online and offline benchmarks and provides this download command:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

Its online benchmark example uses vllm bench serve against /v1/completions. The core documented arguments are:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

Use the complete invocation on the linked page before running it. The documented report includes request throughput, output-token throughput, and total-token throughput.

Use that output for the sustained comparison, while keeping cold latency in a separate record:

  • Verify the zero-worker state, send the fixed request, and record first-response and completion timestamps.
  • Repeat the request while capacity is warm and record the same timestamps.
  • Keep the dataset, model, request shape, and concurrency fixed during throughput comparisons.
  • Copy the provider’s billable start, execution, and idle intervals into the corresponding result.
  • Repeat after each configuration change rather than combining several changes into one run.

Where cold-start comparisons go wrong

The most common comparison errors are procedural:

  • Calling a request cold when a worker was already running.
  • Timing process creation but not actual request readiness.
  • Treating a rejected request as a generation result.
  • Comparing cold latency with steady-state throughput in the same field.
  • Changing weight caching and warm capacity simultaneously.
  • Comparing providers without normalizing their billable setup and idle periods.
  • Treating a longer warm window as free because it removes startup requests.

These steps are compiled from the official documentation linked in the text; nothing was run or measured for it, and the commands and settings should be checked against the current version before use.

Sources