Reduce Serverless GPU Cold Starts for LLM Apps

See how image reuse, model loading, and endpoint readiness shape serverless GPU startup, then tune warm capacity and verify TTFT.

Reduce serverless GPU cold starts by moving weight acquisition out of container boot, retaining only the capacity your traffic needs, and measuring the first streamed response from the client. Modal’s cold-start guide separates queue waiting from first-invocation initialization, while the vLLM benchmark documentation defines time to first token at the client. A reliable llm deploy workflow therefore treats image reuse, model loading, endpoint readiness, and steady-state generation as separate controls, not as an undifferentiated startup metric.

A repeatable llm deploy path for reducing cold starts

Before you change runtime settings, record the current image definition, model and cache location, warm-container policy, request pattern, and client-visible endpoint. This prevents a faster first response caused only by retaining more hardware from being mistaken for better initialization.

Modal caches each image layer—specifically, each Image method call—and a broken cache in one layer causes cascading rebuilds for later layers, so frequently changing layers should come last according to Modal’s Images documentation.

For diagnosis, use the documented build controls deliberately:

MODAL_FORCE_BUILD=1 modal run YOUR_FILE.py

Or set the separate environment variable:

MODAL_IGNORE_CACHE=1

MODAL_FORCE_BUILD forces images attached to the App to rebuild. MODAL_IGNORE_CACHE=1 rebuilds from the top without breaking the image cache for later builds. Neither setting keeps a runtime GPU container warm; they only control image construction.

Build a reusable Modal model path

The official Modal vLLM example runs vLLM in OpenAI-compatible mode, caches Hugging Face weights in a Modal Volume to avoid downloading them on every server start, exposes a FAST_BOOT switch, declares scale-down and startup settings, and deploys with modal deploy vllm_inference.py.

The example creates its Hugging Face cache Volume on demand by name:

hf_cache_vol = modal.Volume.from_name("huggingface-cache", create_if_missing=True)

That line is only the Volume declaration; retain the example’s complete cache mount and environment configuration rather than treating it as a complete server definition.

The boot-mode fragment is:

cmd += ["--enforce-eager" if FAST_BOOT else "--no-enforce-eager"]

With --enforce-eager, vLLM disables Torch compilation and CUDA graph capture to favor faster boots at the cost of generation performance. The example advises FAST_BOOT=True for services that often scale from zero and False when several replicas are normally running.

Its published scale-down and startup settings are:

scaledown_window=15 * MINUTES,
startup_timeout=10 * MINUTES,

After assembling the current official example around the required settings, use its documented deployment command:

modal deploy vllm_inference.py

These are example settings, not guaranteed performance targets. Your model size, initialization work, traffic pattern, and current provider configuration determine the appropriate balance.

Size the warm pool and scaling controls

According to Modal’s cold-start guide, containers boot in about one second, but readiness still waits for global-scope code and all enter methods; the page identifies scaledown_window, min_containers, and buffer_containers, gives a 60-second default idle limit and a configurable range from 2 seconds to 20 minutes for scaledown_window, explains that longer retention bills idle resources, uses min_containers as a scaling floor and buffer_containers for extra active capacity, and notes that moving work into enter shifts rather than removes initialization latency while large file reads should be concurrent where feasible.

Use that separation to make deliberate choices. A scale-down window covers predictable quiet periods, a minimum-container setting protects a persistent floor, and buffer capacity handles bursts while the Function is active. If initialization repeatedly reloads model data, fix that path before paying to hide it with retained capacity.

The RunPod Serverless pricing documentation bills from worker start until it fully stops, rounded up to the next second, and divides billable worker time into start time for container initialization and model loading, execution time, and the idle-timeout period. A cold start therefore adds directly to the period covered by the worker bill.

The RunPod endpoint settings say that one or more Active workers eliminate cold starts but are charged continuously, including while idle; documented defaults include 0 Active workers, 3 maximum workers, a 5-second idle timeout, and FlashBoot enabled, while the page suggests setting maximum workers about 20% above expected peak concurrency and describes FlashBoot as retaining worker state after spin-down, with the greatest benefit for consistent traffic.

Keep the cost trade-off visible without inventing a rate:

estimated_warm_cost = YOUR_BILLABLE_WARM_UNITS × YOUR_CURRENT_QUOTED_RATE

Fill the placeholders from your own quote or invoice and keep the billing unit consistent. This worksheet separates retained-capacity cost from request execution; it is not a provider price.

Separate network readiness from model readiness

A reachable network endpoint and a ready inference container are different states.

The Modal vLLM example states that a Modal Server with no active containers rejects requests with a 503 Service Unavailable status, unlike a Modal Function. Record that response separately from a model error, and define retry behavior as part of your own client contract rather than inventing a retry count from the example.

RunPod’s endpoint configuration options also include queue-delay scaling, which adds workers when requests exceed a waiting threshold—4 seconds by default—and request-count scaling, which targets:

Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)

RunPod documents a scalerValue of 1 for maximum responsiveness and associates this request-count approach with LLM workloads and frequent short requests. These controls address waiting caused by insufficient capacity; they do not remove container or model initialization work.

Verify cold and warm behavior separately

Keep the cold-path observation distinct from steady-state load testing. For the cold path, begin from a controlled state with no ready container, submit one request, and record timestamps on the client:

observed_start_delay = YOUR_FIRST_STREAM_TIME - YOUR_REQUEST_SUBMIT_TIME

Label that result as a cold-start observation. Do not compare it directly with a warmed-server throughput run without showing the different starting conditions.

The vLLM Benchmark CLI documentation uses ShareGPT for online and offline benchmarks, provides the commands below, reports request throughput plus output-token and total-token throughput, measures TTFT from request submission to the first streamed output, defines ITL as gaps between streamed outputs and TPOT as per-request output-token time excluding the first token, sets --request-rate to inf by default, treats an omitted --max-concurrency as unlimited, warns that repeated runs can inflate throughput through prefix-cache reuse, and notes that metric terminology is not standardized across tools.

Download the documented ShareGPT dataset:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

The documented online command points at a running vLLM server:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

Use the default inf request rate only when maximum-throughput behavior is the intended test. For controlled load, provide a finite --request-rate; set --max-concurrency when you need a cap. Record server state and benchmark-run identity so prefix reuse is not mistaken for an improvement caused by your deployment change.

Where startup usually goes wrong

Use separate rows for image build, weight acquisition, container readiness, queue waiting, and first streamed output. Then check for these common mistakes:

Changing image-cache controls and interpreting the result as a runtime speedup.
Adding warm capacity before confirming that the intended model cache is being used.
Treating a reachable URL as proof that the model is ready for inference.
Comparing a one-request cold sample with warmed-server throughput without labeling both conditions.
Repeating a benchmark without recording server state, request pattern, and run identity.

Change one control at a time, then compare equivalent client-side measurements. These steps are compiled from the official documentation linked in the text and should be checked against the current version before use; nothing here was run or measured for this article.