AI DeploySelf-hosted LLMGet it deployed

Self-Hosted vs Serverless LLM Inference Deployment

Compare dedicated RunPod Pods with Serverless endpoints, showing that Pod compute and storage are billed per second while Serverless bills worker runtime.

When choosing a self hosted llm, use a dedicated GPU runtime when you need control over the application environment and runtime lifecycle; use serverless inference when requests should be handled by endpoint workers. On RunPod, Pods bill compute and storage per second, require at least one hour of credits for the selected configuration before on-demand deployment, and can continue charging storage after a stop; Serverless bills from worker start until the worker fully stops. Treat model-server flags and request payloads as a separate compatibility task tied to the selected model and runtime.

A self hosted llm: what changes between Pods and endpoints?

A Pod is the dedicated-runtime option: the GPU is rented, but you decide when the environment and inference service run. Serverless is the endpoint option: workers start, process requests, and stop according to endpoint configuration.

Choose a Pod for a predictable, long-lived session that your team wants to administer. Choose Serverless for request-driven traffic when startup, scaling, and billed idle time fit your operating model. The decision is therefore about lifecycle and cost control, not only accelerator capacity.

Self-hosted Pod path: provision, verify, and control cost

Prepare the deployment

Before provisioning, write down the inputs needed to reproduce and verify the deployment:

  • The model artifact, weight format, and intended runtime.
  • The exact vLLM startup command, flags, listening port, health check, and request schema from the current official documentation.
  • The GPU type and count required by that model configuration.
  • The container image, Pod name, persistent storage decision, and current billing quote.
  • The concurrency and request-duration pattern you expect, especially if you are comparing this setup with Serverless.

Do not treat generic startup examples as verified configuration. A Pod-provisioning command creates infrastructure; it does not prove that the model fits the GPU, loads successfully, or exposes a working inference API.

Create and control a Pod

The Pod management guide defines the console, CLI, REST, and lifecycle controls. Its documented console sequence is:

  1. Select the GPU type.
  2. Name the Pod.
  3. Optionally select a template.
  4. Set the GPU count.
  5. Click Deploy On-Demand.

The documented CLI form is:

runpodctl pod create --name YOUR_VALUE --gpu-id "YOUR_VALUE" --image "YOUR_VALUE"

The documented REST creation method, endpoint, and authorization header are:

curl --request POST \
  --url https://rest.runpod.io/v1/pods \
  --header 'Authorization: Bearer YOUR_VALUE'

That REST snippet establishes the request method, URL, and header; it is not a complete production request because the current request-body schema must still be obtained from the API reference.

Storage behavior determines whether stopping is sufficient. Stopping releases the GPU, preserves data in /workspace, and clears the container disk. Terminating permanently deletes data that is not stored on a network volume and stops the stopped-Pod volume-disk charge. If the environment must survive a restart, place resumable data in the appropriate volume rather than relying on container storage.

Verify the Pod in layers

Do not use Pod creation success as the only acceptance test. Check the deployment in three layers:

  1. Infrastructure: Confirm that the intended GPU allocation is present and the Pod has reached its expected running state.
  2. Runtime: Start the model with the exact documented command and call that runtime’s documented readiness check.
  3. Inference: Send the smallest valid request using the documented path and schema, then retain the response and relevant logs.

This sequence distinguishes an infrastructure problem from an incompatible model, a failed model load, and an invalid request contract without inventing an endpoint that the infrastructure documentation does not define.

Calculate the Pod cost

Read the current Pod pricing and plan terms as a line-item sheet rather than looking only at a GPU rate. Compute and storage are charged per second; container and volume disks are charged per second, while network volumes are charged hourly. Data ingress and egress have no listed fee, and storage is not charged while the host is unavailable.

Build the estimate with rates taken from your quote or bill:

estimated_pod_cost =
  GPU_COUNT * YOUR_VALUE * BILLABLE_SECONDS
  + CONTAINER_GB * YOUR_VALUE * BILLABLE_SECONDS
  + VOLUME_GB * YOUR_VALUE * BILLABLE_SECONDS
  + NETWORK_VOLUME_GB * YOUR_VALUE * BILLABLE_HOURS

Replace every YOUR_VALUE with the corresponding provider rate. Include stopped periods in the relevant storage variables because stopping a Pod does not mean every storage charge has ended.

RunPod savings plans are 3- or 6-month upfront commitments, are non-refundable, and have fixed expiration dates. After a Pod stops, the plan applies to the next deployment using the same GPU type. Treat those as contract terms and verify them in the current page and quote before committing.

If the dedicated runtime uses AWS, include resources outside the instance line. AWS documents that a stopped EC2 instance is not billed for instance usage, but EBS volumes and Elastic IP addresses can incur charges regardless of instance state. Each stopped-to-running transition also starts a new billing period with a one-minute minimum, according to the EC2 instance lifecycle documentation.

Serverless LLM inference: configure endpoint workers

Price the complete worker lifetime

The Serverless pricing rules count billable time from worker startup until the worker fully stops, rounded up to the next second. That interval includes container initialization and model loading, request execution, and the idle-timeout period after a request finishes.

Calculate each worker configuration separately and then sum the result:

estimated_serverless_cost =
  TOTAL_BILLABLE_WORKER_SECONDS * YOUR_VALUE

TOTAL_BILLABLE_WORKER_SECONDS must include startup, execution, and billed idle time. A comparison that counts only token-generation time will not represent the documented billing clock.

Configure scaling and hard limits

The documented endpoint settings provide these defaults:

  • Active workers: 0.
  • Maximum workers: 3.
  • Idle timeout: 5 seconds.
  • Execution timeout: 600 seconds, with a documented range from 5 seconds to 7 days.
  • Job TTL: 24 hours, counted from submission.
  • FlashBoot: enabled.

Setting active workers to one or more keeps them warm and incurs charges continuously, including while idle. That removes cold starts but trades them for ongoing worker charges.

For request-count scaling, the documented worker target is:

Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)

RunPod recommends a scalerValue of 1 for maximum responsiveness and describes this scaling method as suitable for LLM workloads and frequent short requests. It also suggests setting maximum workers about 20% above expected peak concurrency. These are capacity settings, not billing guarantees; calculate each worker type and idle period separately.

Endpoints with no requests for three days have their maximum workers reduced to two, and after seven days it is reduced to zero. Account for that behavior before treating an old endpoint as unchanged production capacity.

Where deployments usually go wrong

  • OCI runtime create failed: The Pod management guide’s CUDA filter guidance directs you to Additional filters → CUDA Versions so deployment can select compatible machines.
  • A restarted Pod has no GPU: RunPod’s restart troubleshooting documentation explains that a stopped Pod remains tied to its physical machine while its released GPU may be assigned elsewhere. A restart can therefore find zero available GPUs.
  • The Pod runs but the model does not: Recheck the current model and runtime requirements, exact startup command, GPU fit, and readiness check. Infrastructure creation success does not validate model compatibility.
  • Queue time consumes the job allowance: The documented job TTL starts at submission, not when execution begins. A request that waits in the queue has less execution time available.
  • Idle cost is misunderstood: Decide whether the latency benefit of active workers or the cost of billed idle timeout is acceptable before sending production traffic.

Make the decision from the workload

Choose the dedicated Pod path when the application needs a sustained runtime, predictable administration, and explicit control over storage and process lifecycle. Accept responsibility for GPU availability, stopped storage, persistence, and the complete model-server configuration.

Choose the Serverless path when requests arrive through an endpoint and demand varies enough to justify request-driven workers. Accept that startup, model loading, scaling, idle timeout, execution timeout, and job TTL all become cost and failure controls.

Before choosing either path, obtain the current model-server documentation and complete the provider-specific cost worksheet. If you cannot fill in the startup command, request schema, or billable line items, the deployment is not yet ready for production.

These steps are compiled from the official documentation linked in the text; no command was run and no performance result was measured for this article, so check them against the current version before use.

Sources