AI DeploySelf-hosted LLMGet it deployed

Set Up Reserved GPU Capacity for LLM Serving

Reserve RunPod active workers or Modal warm containers, then plan LLM serving around documented billing, scaling, and cold-start behavior.

Set up reserved GPU capacity for llm hosting by choosing a provider-documented warm-capacity floor, applying it to the serving deployment, and checking startup behavior and billing before rollout. Do not assume a universal “reserve a GPU” command: the exact setting, billable unit, and meaning of retained capacity come from the provider’s current deployment controls.

Plan llm hosting around retained capacity

A capacity floor can remove a scale-from-zero step, but it does not by itself establish model fit, throughput, dedicated hardware, or a contractual reservation. Check the provider’s current service terms when those distinctions matter.

Before changing a deployment, record:

  • The exact model identifier and where its weights are stored.
  • GPU memory needed for weights, KV cache, and working memory.
  • Expected peak concurrent requests and likely bursts.
  • How long capacity should remain available during idle periods.
  • The provider’s current billing unit, rate, and covered setup or idle phases.

Keep these as planning inputs rather than performance guarantees. In particular, a maximum request setting is not a throughput measurement, and a warm container is not automatically dedicated hardware.

Prepare the RunPod vLLM worker

Start with the documented RunPod vLLM worker environment variables:

MODEL_NAME=YOUR_VALUE
MAX_CONCURRENCY=YOUR_VALUE

The RunPod vLLM environment-variable reference says MODEL_NAME accepts a Hugging Face repository ID or local filesystem path and defaults to facebook/opt-125m; it also sets MAX_CONCURRENCY to 30 by default, defines that setting as a RunPod-side per-worker request limit, and explicitly distinguishes it from a vLLM engine argument.

Use a separate worksheet for capacity arithmetic:

GPU_MEMORY_NEEDED_GB =
  MODEL_WEIGHT_MEMORY_GB + KV_CACHE_GB + ACTIVATION_HEADROOM_GB

CONFIGURABLE_REQUEST_CEILING = MAX_WORKERS × MAX_CONCURRENCY

These symbols are worksheet fields, not additional provider settings. The memory formula is only a starting point for a single-worker layout; follow the model runtime’s current placement requirements when weights are sharded. Likewise, the request ceiling is derived from configuration and does not predict how many requests the GPU can process concurrently under a real workload.

Reserve RunPod workers and connect the client

Use the RunPod endpoint settings as the reservation baseline: active workers are the continuously billed warm minimum and a value of 1+ eliminates cold starts, while max workers is the concurrent-worker ceiling; the documented defaults are 0 active workers, 3 max workers, a 5-second idle timeout, a 600-second execution timeout, a 24-hour job TTL, and FlashBoot enabled; request-count scaling targets Math.ceil((requestsInQueue + requestsInProgress) / scalerValue), with a scaler value of 1 recommended for maximum responsiveness and LLM or frequent short-request workloads.

Set the active-worker count to the floor your traffic requires, then choose the maximum-worker setting separately. Do not treat the active floor as the endpoint’s concurrency ceiling.

After deploying the vLLM worker, configure the compatible client with the base URL documented by RunPod:

https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1

The RunPod OpenAI API compatibility documentation publishes that URL for the vLLM worker. Replace ENDPOINT_ID with the actual endpoint identifier, then use a normal request through the same client integration you will use in service. A valid response checks the request path; it does not prove the reserved worker count, concurrency, or output speed.

Keep Modal containers warm

Use the documented Modal decorator arguments as a configuration template:

@app.function(
    gpu=["H100", "A100-40GB:2"],
    min_containers=YOUR_VALUE,
    scaledown_window=YOUR_VALUE,
    buffer_containers=YOUR_VALUE,
)
def serve():
    pass

Replace pass with your serving implementation and fill each YOUR_VALUE from your capacity plan. The Modal GPU guide documents ordered GPU fallback lists such as the one shown, multiple GPUs by appending :n, larger waits when requesting more than 2 GPUs, placement of all GPUs in one container on one physical machine, and automatic substitution of an gpu="H100" request with an H200 at the same cost unless the request uses gpu="H100!".

The Modal cold-start guidance defines min_containers as a floor that prevents the Function from scaling to zero, distinguishes buffer_containers as extra capacity while the Function is active, documents a 60-second default scaledown_window settable from 2 seconds to 20 minutes, warns that longer retention reduces cold-start chances while idle resources remain billable, recommends downloading weights ahead of time and reading large files concurrently, and notes that global initialization and modal.enter methods must finish before a container is warm.

Treat these as warm-container controls. If exact GPU identity or dedicated hardware is required, verify that separately rather than inferring it from min_containers.

Set a Replicate deployment minimum

The Replicate billing documentation says public models are billed only for active processing and use a shared hardware pool by default, while deployments are billed for setup, idle, and active time; it also says a deployment’s minimum instance count defaults to 0 but can be customised for deployments.

Set a nonzero minimum through Replicate’s current deployment controls if retained model instances are required. The cited billing page does not name the configuration key, so do not invent one; check the current deployment documentation for the exact field. Include setup, idle, and active online time in the quote before treating the deployment as reserved capacity.

Verify each part before rollout

Check the resolved deployment configuration, not only the source file:

  • RunPod: Confirm that active workers are nonzero, max workers remains a separate ceiling, and MODEL_NAME and MAX_CONCURRENCY contain the intended values.
  • Modal: Confirm the GPU or fallback list, min_containers, buffer_containers, and scaledown_window match the deployment plan.
  • Replicate: Confirm that the deployment minimum is nonzero and that the quote covers its online phases.
  • Model readiness: Make the model artifact available before the serving path expects it, rather than treating a configured floor as proof that every initialization step has completed.
  • Request path: Send an ordinary request through the production client path and confirm that the intended model returns the expected response format.
  • Billing path: Compare the first usage record with the current quote’s unit and retained-capacity settings.

One successful request is only a functional check. It does not establish throughput, latency, capacity headroom, or behaviour under a burst.

Calculate the reserved-capacity cost

On the provider’s current pricing or billing page, read the retained-capacity unit, its floor, the applicable rate, which setup or idle phases are billable, and every separate item shown on the quote or bill. Then fill this calculation from your own figures:

RUNPOD_COST_ESTIMATE =
  ACTIVE_WORKER_BILLABLE_SECONDS × ACTIVE_WORKER_RATE
  + OTHER_ITEMS_ON_QUOTE_OR_BILL

MODAL_COST_ESTIMATE =
  SUM(RESOURCE_BILLABLE_SECONDS × RESOURCE_QUOTED_RATE)
  + OTHER_ITEMS_ON_QUOTE_OR_BILL

REPLICATE_COST_ESTIMATE =
  ONLINE_INSTANCE_SECONDS × INSTANCE_RATE
  + OTHER_ITEMS_ON_QUOTE_OR_BILL

Replace each worksheet variable with a value from the current provider quote or usage record. The cited documentation does not establish one universal rate, so do not substitute a remembered or guessed currency amount. Request ceilings and configured container counts can help define the reservation, but only the provider’s billable unit and quote determine the estimate.

Common failure points

  • Confusing the floor with the maximum: Set the retained minimum first, then review the separate maximum, buffer, or autoscaling controls.
  • Assuming exact GPU identity: Review any fallback count, model substitution, physical-machine placement, and upgrade behavior before selecting a GPU value.
  • Putting concurrency in the wrong layer: Apply RunPod’s documented worker-side limit to the RunPod setting rather than treating it as a vLLM engine argument.
  • Forgetting model initialization: Confirm that weights and initialization are available before a container is treated as ready.
  • Ignoring retained idle time: Include every documented billable phase before enabling a warm floor.
  • Calling a basic response a benchmark: Validate capacity separately; no result here was run or measured.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources