Choose a Serverless GPU for LLM Inference

Compare GPU memory, startup controls, scaling limits, model support, and regional availability before selecting a serverless inference backend.

For llm hosting, choose a backend by applying hard exclusions first, then matching GPU memory, startup behavior, scaling controls, request limits, billing rules, and regional availability to the model and traffic. A GPU name alone does not show that an endpoint can load the model or meet the expected operating pattern. Use the checks below as a go-or-exclude record, not as a vendor ranking.

Start with hard exclusions

Amazon SageMaker Serverless Inference explicitly does not support GPUs, so the current Amazon SageMaker Serverless Inference documentation rules it out for a GPU-based deployment.

Cloud Run documents a GPU path: its GPU support documentation lists the NVIDIA RTX PRO 6000 Blackwell with 96 GB of VRAM and the NVIDIA L4 with 24 GB, allows GPU services to scale to zero, and states that GPU instances with pre-installed drivers start in approximately 5 seconds. That startup figure covers the instance becoming available to container processes; it does not include loading the model.

Do not treat scale-to-zero as equivalent to permanent readiness. It changes the idle operating model, but the first request may still depend on container startup and model loading.

Match GPU memory to the inference workload

Start with usable accelerator memory rather than a product label. NVIDIA lists 80 GB of memory for the H100 SXM and 94 GB for the H100 NVL in its H100 specifications. Those figures describe the accelerators themselves; they do not prove that a particular serverless endpoint exposes either configuration.

For each candidate, record the following before selecting it:

Model footprint: the weights, numerical format, and any additional model artifacts that must be resident on the GPU.
Runtime demand: reserve room for the KV cache, temporary tensors, framework workspace, and request concurrency.
VRAM ceiling: confirm the exact GPU memory available to the selected service configuration rather than relying on the GPU family name.
Model compatibility: verify that the provider documents your model architecture and inference runtime, not merely that it offers GPUs.
Regional availability: confirm that the selected GPU type is offered in the Region where the endpoint will run.

If the estimated memory requirement does not fit, changing concurrency settings will not fix the problem. Select a larger-memory configuration, change the model representation, or evaluate a different backend.

Decide what startup and billing model you need

Serverless does not mean that initialization is free or instant. RunPod's Serverless pricing documentation says billing starts when a worker starts and ends when it fully stops, rounded up to the next second. It includes container initialization and model loading in start time, divides billable worker time into start, execution, and idle phases, and gives the idle timeout a default of 5 seconds.

Use that billing boundary to construct a quote:

BILLED_WORKER_SECONDS = ceil(START_SECONDS + EXECUTION_SECONDS + IDLE_SECONDS)
ESTIMATED_WORKER_CHARGE = BILLED_WORKER_SECONDS × YOUR_RATE_PER_BILLED_SECOND
ESTIMATED_TOTAL = SUM(ESTIMATED_WORKER_CHARGE_ACROSS_WORKERS)

Enter YOUR_RATE_PER_BILLED_SECOND from the provider’s current quote. No rate is supplied here, and the formula should not be transferred to another provider whose billing units have not been documented.

This is the llm hosting trade-off: a backend that scales away unused capacity can reduce idle consumption, while maintained capacity trades continuous charges for faster availability. Base that decision on your traffic pattern rather than assuming either side is always preferable.

Configure a Modal GPU explicitly

Modal attaches GPUs through the gpu argument on a Function decorator. Its GPU acceleration guide requires a valid payment method for GPU use, documents the identifiers T4, L4, A10, L40S, A100, A100-40GB, A100-80GB, RTX-PRO-6000, H100/H100!, H200, B200/B200+, and B300, and explains that multiple GPUs are requested by appending :n.

The documented multi-GPU expression is:

gpu="H100:8"

Treat the identifier list as configuration syntax, not as proof that every option is available in every Region or supports every model. Check the current GPU page for regional inventory, then verify model and runtime compatibility separately before deployment.

Set RunPod scaling and timeout controls deliberately

RunPod's endpoint settings document request-count scaling with this target:

Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)

The documentation recommends request-count scaling for LLM workloads or frequent short requests and identifies scalerValue set to 1 as the choice for maximum responsiveness. It gives queue-delay scaling a default threshold of 4 seconds and suggests setting maximum workers about 20% above expected peak concurrency.

The documented endpoint defaults are 0 active workers, 3 maximum workers, a 5-second idle timeout, a 600-second execution timeout, a 24-hour job TTL, and enabled FlashBoot. Active workers remain warm and incur continuous charges, including while idle. FlashBoot retains worker state after a worker spins down and is described as most effective when traffic is steady enough for workers to cycle between active and idle states.

Two timeout rules can otherwise surprise you. The execution timeout can range from 5 seconds to 7 days; when it is exceeded, the job fails and the worker stops. The job TTL is a hard limit that starts when the request is submitted rather than when execution begins, so queue time reduces the time available for inference.

Verify the choice before deployment

Create one row per candidate and require evidence for every hard requirement:

[ ] The exact GPU identifier and available VRAM are documented.
[ ] The model architecture, numerical format, and runtime are supported.
[ ] The GPU configuration is available in the intended Region.
[ ] Container startup and model loading are included in your latency assessment.
[ ] The scaling formula uses your expected queued and active request pattern.
[ ] Execution timeout and overall job lifetime cover queueing as well as inference.
[ ] Start, execution, idle, and maintained-capacity charges are included in the calculation.
[ ] Any capacity recommendation is backed by your own application verification rather than a GPU name or generic capacity tier.

Exclude a candidate when a required capability is undocumented or incompatible. If more than one option passes, compare their startup, scaling, timeout, regional, and billing behavior using the same workload assumptions.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.