AI DeploySelf-hosted LLMGet it deployed

Measure Cold Starts Across Serverless LLM Platforms

Build a repeatable cold-start test that separates observable startup phases from queue and response latency across serverless platforms.

Measure cold starts across serverless LLM platforms by establishing the most specific no-ready state each provider documents, sending the same logical request, and recording client timestamps with any available platform phase markers. For llm hosting, keep queueing, startup, model loading, and response latency separate when observable; when those boundaries are unavailable, label the result an end-to-end proxy rather than a pure cold-start measurement.

Control variables for llm hosting benchmarks

A repeatable run begins with a manifest, not an ad hoc stopwatch. For each test cell, record:

  • The platform and exact function, endpoint, or deployment identifier.
  • The model identifier and exposed revision.
  • The request body, streaming mode, and output controls.
  • The container image identifier or digest and the runtime and dependency lock.
  • The GPU class and location setting, if exposed.
  • The documented model-loading path.
  • Worker or scaling controls.
  • The timestamp source, cold-state evidence, and trial outcome.

Hold these values constant across matched cells. If a platform does not expose a model revision, image digest, GPU class, or state indicator, record that limitation instead of guessing.

A worksheet can use this structure:

{
  "platform": "YOUR_VALUE",
  "object_id": "YOUR_VALUE",
  "model_revision": "YOUR_VALUE",
  "image_digest": "YOUR_VALUE",
  "gpu_class": "YOUR_VALUE",
  "model_loading_path": "YOUR_VALUE",
  "scale_config": "YOUR_VALUE",
  "trial_plan": "YOUR_VALUE"
}

This is a worksheet schema, not a provider API payload.

Define the measurement before sending traffic

Use a monotonic client clock and capture the request-send timestamp immediately before invoking the deployed object. For streaming responses, capture the first-token and final-response timestamps. Keep wall-clock timestamps as well so another operator can audit the run.

Use these worksheet definitions:

request_to_first_token = first_token_timestamp - request_send_timestamp
request_to_final = final_response_timestamp - request_send_timestamp
request_to_ready = provider_ready_timestamp - request_send_timestamp

Populate request_to_ready only when the provider supplies a documented readiness event. Never backfill it from the first token. Without streaming, retain the final-response measurement but leave the first-token field unavailable rather than reconstructing it.

NVIDIA’s GenAI-Perf README says the tool is no longer receiving new feature development and directs new performance benchmarking to AIPerf. It defines time to first token as the time between sending a request and receiving its first response. That makes TTFT a useful workload measurement, but it does not identify the provider’s startup boundary by itself; follow the documentation’s redirection before adding a new load generator.

Establish each platform’s cold state

Use the provider’s own state definition, then preserve the evidence that established it. Do not substitute an undocumented sleep period for state verification.

Modal

Modal’s cold start performance guide defines a cold start as occurring when no ready container exists and Modal must start one. It attributes the added latency to waiting for a container to become warm and first-invocation initialization; global-scope code and modal.enter methods must finish before the container is considered warm, and inputs are not routed before initialization completes.

The same guide recommends downloading model weights ahead of time and identifies scaledown_window, min_containers, and buffer_containers as controls for warm containers. It also warns that a longer scaledown_window can reduce subsequent cold starts while leaving resources billable during idle periods. Record all of these conditions without changing them midway through a comparison.

RunPod

RunPod defines a cold start in its Serverless overview as the interval from an endpoint with no running workers receiving a request until a worker is fully warmed and ready to handle it.

RunPod’s endpoint settings state that setting active workers to 1+ eliminates cold starts and incurs continuous charges while idle. They also state that FlashBoot retains worker state after spin-down. Keep the active-worker arrangement and FlashBoot setting fixed within each test cell; use active workers only as a separately labelled warm control.

Replicate

Replicate’s billing documentation says a model without demand scales to its configured minimum instance count, which is 0 by default and can be customized for deployments. It also says an instance remains idle for a few minutes after a request gap rather than shutting down immediately.

The same documentation says public models share a hardware pool and queue, with only active request-processing time charged; setup and idle time are free. It says most private models run on dedicated hardware without a shared queue, with fast-booting fine-tunes identified as the exception. For Replicate, retain request-to-first-response and request-to-final as end-to-end fields unless a documented provider phase boundary is available. Record whether the model is public or private and preserve the configured minimum-instance setting.

Run matched trials

Before each trial:

  • Capture the platform state and the evidence showing that it is cold.
  • Send the same logical request used for every matched cell.
  • Record raw timestamps, response status, output settings, and any documented phase events.
  • Restore the intended state before collecting another cold observation.
  • Choose the trial plan before collection and preserve unsuccessful observations with their reported status.

A completed response does not by itself prove that the trial was cold. Classify the result explicitly:

measurement_scope = "provider_cold_start" | "end_to_end_proxy"
valid_trial = request_completed AND state_evidence_present

These are worksheet classifications, not provider fields. If state evidence is incomplete, retain the trial but keep it outside a pure cold-start comparison.

Check the measurement worked

Before comparing results:

  • Recalculate every duration from the raw timestamps.
  • Confirm that timestamps came from the same client clock and unit.
  • Verify the model, image, GPU class, and scale configuration against the manifest.
  • Check that unavailable startup or model-loading phases remain explicitly missing.
  • Separate warm controls from cold observations.
  • Preserve failures, timeouts, and incomplete responses.
  • Reject matched comparisons when several configuration fields changed together.

A proxy remains a proxy after filtering or summarization. Do not convert an end-to-end measurement into a provider startup claim merely because the run completed successfully.

Where comparisons usually break

  • Metric confusion: TTFT includes everything between sending the request and receiving its first response. It may be useful, but it is not automatically cold-start time.
  • Queue blindness: A shared queue can sit inside the client-observed interval. Do not attribute the whole interval to container startup or model loading without a documented boundary.
  • State contamination: A previous request may leave an object ready. Verify the intended state again before labelling the next observation.
  • Configuration drift: Changing weights, image contents, GPU class, worker count, or scale controls invalidates a matched comparison.
  • Billing confusion: RunPod’s Serverless pricing documentation says billing starts when a worker starts and continues until it fully stops, rounded up to the next second. It includes container initialization and model loading in start time, and idle timeout is also billable. Keep billable duration in a separate field; it is not a substitute for client latency.
  • Selective reporting: Do not retain only successful or fast observations. Publish the distribution, chosen summary, and failures so readers can see tail behavior and state contamination.

Report matched results, not a winner

Each result row should identify the platform object, model revision, image, GPU class, scale configuration, cold-state evidence, measurement scope, raw-derived durations, billable duration when relevant, and failed trials. Compare only rows with matching workload and configuration variables.

Keep provider-defined cold-start observations separate from end-to-end proxies. Do not place both in the same ranking or claim that one platform is faster when their measurement boundaries differ. A repeatable result is one that another operator can rerun from the manifest, raw timestamps, documented state evidence, and current product documentation.

These steps are compiled from the official documentation linked in the text; nothing was run or measured for it, and the current versions should be checked before use.

Sources