Self-Hosted LLM Serverless Inference for Variable Traffic

Deploy an LLM with serverless workers, choose warm-capacity controls for changing traffic, verify requests, and calculate usage-based cost.

For variable traffic, self hosted llm deployments can provide serverless inference when workers start on demand and warm capacity is governed by explicit limits. RunPod’s endpoint settings document queue-delay and request-count scaling, while Modal’s cold-start guidance separates queueing for a warm container from first-invocation initialization. The practical choice is not “always warm” versus “always cheap”; it is a documented boundary between burst capacity, cold-start exposure, idle duration, and request retention.

Set boundaries for a self hosted llm deployment

With rented GPUs, your team can control the model, worker image, request handler, endpoint settings, access policy, and scaling rules while the provider supplies the execution layer. That boundary matters because a serverless design is not a permanently allocated machine: the worker lifecycle becomes part of the application design.

Before deployment, write down:

The exact model identity and the location from which the worker will load it.
The request operation your application will call and how it will retrieve results.
Expected concurrency, burst duration, and the point at which added latency becomes unacceptable.
Which capacity may remain warm, how long it may remain idle, and how usage will be checked against current provider rates.

This operating contract prevents a scaling decision made during an incident from becoming an undocumented permanent cost.

Configure RunPod for serverless inference

The following defaults and controls come from RunPod’s endpoint settings.

Control	Documented default	Operational effect
Active workers	0	Sets the minimum number of workers kept warm. One or more active workers eliminates cold starts, but active workers are charged continuously, including while idle.
Max workers	3	Sets the maximum concurrent workers. RunPod suggests setting it about 20% above expected peak concurrency.
Idle timeout	5 seconds	Keeps a worker running after a request before scaling it down; this idle period is billed.
Queue-delay threshold	4 seconds	Adds workers when requests wait longer than the threshold.
Execution timeout	600 seconds	Fails the job and stops the worker when the limit is exceeded.
Job TTL	24 hours	Acts as a hard limit measured from submission, not from execution start.
FlashBoot	Enabled	Retains worker state after spin-down. RunPod describes it as most effective for endpoints with consistent traffic and frequent worker cycling.

For request-count scaling, the worker target is:

Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)

RunPod recommends request-count scaling for LLM workloads and frequent short requests, and identifies a scalerValue of 1 as the setting for maximum responsiveness. Treat this as the endpoint’s desired worker count: it is different from both the maximum-worker ceiling and the per-worker vLLM concurrency control.

Configure the vLLM worker before opening traffic

Set the exact worker keys documented in RunPod’s vLLM environment variables:

MODEL_NAME
MAX_CONCURRENCY

MODEL_NAME accepts a Hugging Face repository ID or a local filesystem path for model weights. Its documented default is facebook/opt-125m, so set it explicitly when the endpoint is intended to serve a different model.

MAX_CONCURRENCY has a documented default of 30. It limits concurrent requests per vLLM worker on the RunPod side and is not a vLLM engine argument. Configure the model and engine first, then set this worker-level limit rather than assuming that increasing it changes the vLLM runtime configuration.

Send and verify the documented endpoint

The synchronous request target and credential behavior are defined in RunPod’s Send API requests documentation. Send requests to the documented endpoint pattern with a Bearer API key:

https://api.runpod.ai/v2/ENDPOINT_ID/runsync

Replace ENDPOINT_ID with the deployed endpoint identifier. Choose the operation according to how the application retrieves results:

/runsync waits for completion. Its default wait is 90 seconds, adjustable through ?wait= from 1000 to 300000 milliseconds. Results remain available for 1 minute and for a maximum of 5 minutes.
/run processes the job asynchronously. Retrieve output through /status; results remain available for 30 minutes after completion.
For /run, the documented per-endpoint and per-operation allowance is 1000 requests per 10 seconds and 200 concurrent requests. Excess requests receive HTTP 429.

Verify the integration in this order:

Build a request body that matches the currently deployed model and worker schema.
For /runsync, confirm that the expected output arrives within the selected wait and retention window.
For /run, follow /status until completion and retrieve the result before its retention window closes.
Submit a controlled traffic burst and check whether worker additions follow the configured queue or request-count rule.

An endpoint responding to authentication is not enough: successful verification requires the expected model output through the selected operation.

Tune Modal capacity for bursts without hiding idle cost

Modal’s cold-start guidance defines a cold start as occurring when no ready container exists. It identifies two separate sources of delay: waiting for a warm container and initialization performed on the first invocation. Global-scope code and modal.enter methods must finish before the container is considered warm, so moving work into that hook shifts the latency into warm-up rather than removing it.

Three documented controls determine how many containers remain warm:

scaledown_window controls the maximum idle period. Its default is 60 seconds, and the documented range is 2 seconds to 20 minutes. A longer window reduces subsequent cold starts, but idle resources remain billable.
min_containers establishes a floor so the Function does not scale to zero.
buffer_containers provisions additional idle containers while the Function is active, which is relevant to burst handling.

For initialization, Modal recommends downloading model weights ahead of container startup and reading multiple large files concurrently when feasible.

GPU selection and fallback behavior are documented in Modal’s GPU acceleration guide. The gpu argument accepts an ordered fallback list; this is the guide’s example:

@app.function(gpu=["H100", "A100-40GB:2"])

Modal tries the preferred H100 before falling back to two A100-40GB GPUs. The guide identifies L40S as its starting recommendation for inference and notes that it provides 48 GB of GPU memory. It also warns that requests for more than two GPUs usually face longer waits and that all GPUs in one container sit on the same physical machine.

Check exact-hardware assumptions before deployment. For example, an H100 request may run on an H200 at the same cost; H100! opts out of that automatic upgrade.

Calculate the bill from intervals, not guesses

RunPod Serverless pricing states that billing runs from worker start until the worker fully stops, rounded up to the next second. It divides billable worker time into start, execution, and idle-timeout phases.

Use the current per-second rate from the provider’s pricing page and complete this worksheet separately for each worker interval:

estimated_total = SUM(
  ROUND_UP(
    YOUR_START_SECONDS
    + YOUR_EXECUTION_SECONDS
    + YOUR_IDLE_SECONDS
  ) * YOUR_PER_SECOND_RATE
)

Count one continuous active-worker period once rather than splitting the same warm interval into several requests. Add any other line items shown in your own quote or bill, and replace YOUR_PER_SECOND_RATE whenever the provider changes its rates. Modal’s idle GPU reservation and residual memory occupancy should likewise be evaluated against its current billing documentation rather than copied into the RunPod formula.

Where variable-traffic deployments usually go wrong

The wrong model answers. Verify the configured model identity instead of assuming the endpoint name determines which weights were loaded.
The scaling target exceeds the worker ceiling. Compare the request-count result with the endpoint’s maximum workers before increasing concurrency.
Cold-start time is treated as one problem. Check queue waiting separately from global initialization and modal.enter completion.
Execution timeout and job TTL are confused. Model runtime limits and submission-to-expiration limits answer different questions.
The client misses result retention. Confirm the selected operation, wait behavior, and retrieval schedule before integrating it into a retry policy.
Warm capacity outlives the burst. Revisit active workers, idle timeouts, and warm-container settings whenever traffic patterns change.

Production sign-off

Before production, you should be able to identify the loaded model, the worker-scaling trigger, the maximum worker count, what remains warm, the request operation, the result-retention window, and the current billing rate without guessing. If any answer is unclear, fix that operating boundary before increasing traffic.

These steps are compiled from the official documentation linked in the text, were not run for this article, and should be checked against the current version before use.