Deploy an LLM with RunPod Serverless GPUs

Deploy a vLLM model as a RunPod Serverless endpoint, configure scaling and timeouts, then validate its OpenAI-compatible API.

An llm deploy on RunPod combines a vLLM worker, an endpoint policy, and a request path you can verify. The RunPod vLLM worker overview says the worker can be deployed from RunPod Hub or customized from the runpod-workers/worker-vllm repository. Configure the model and concurrency first, then apply endpoint controls and test the interface your application will use.

Prepare the model and deployment inputs

Before opening the deployment controls, settle the inputs that determine whether the endpoint is usable:

The exact model reference you intend to configure.
A GPU supported by the model’s current requirements.
Your expected peak concurrency and normal request duration.
Whether occasional worker initialization is acceptable or you need permanently warm capacity.

Do not infer GPU compatibility from the worker type or model name alone. Check the model project’s current documentation, then confirm that the selected RunPod GPU configuration satisfies its requirements. The RunPod vLLM documentation does not establish a universal model-to-GPU mapping.

Deploy the vLLM worker

Choose a documented worker route

Use RunPod Hub when the standard vLLM worker fits the deployment, or use the runpod-workers/worker-vllm repository when you need a custom worker. Both routes are documented in the RunPod vLLM worker overview.

Console labels and deployment fields can change, so use the current page for the exact image, template, and input sequence rather than relying on a copied UI path.

Set the worker environment

The RunPod vLLM environment-variable reference defines these relevant settings:

Setting	Value to enter
`MODEL_NAME`	`YOUR_VALUE`
`MAX_CONCURRENCY`	`YOUR_VALUE`

MODEL_NAME accepts a Hugging Face repository ID or a local filesystem path for the model weights. Its documented default is facebook/opt-125m, so replace that value unless it is deliberately the model you want.

MAX_CONCURRENCY defaults to 30 and limits concurrent requests handled by each vLLM worker on RunPod’s side. It is not a vLLM engine argument, so do not move it into the engine configuration after the image tag. Record both values with the endpoint configuration so later changes remain auditable.

Apply the endpoint policy

The RunPod endpoint settings give you a documented baseline: 0 active workers, 3 maximum workers, a 5 s idle timeout, a 600 s execution timeout, a 24 h job TTL, and FlashBoot enabled.

Active workers remain warm and incur charges continuously, including while idle. The execution timeout can range from 5 s to 7 days; when it is exceeded, the job fails and the worker stops. The TTL clock starts when a job is submitted, not when execution begins, which matters when requests wait in the queue.

For request-count scaling, RunPod targets:

Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)

RunPod recommends this scaling method for LLM workloads and frequent short requests, and identifies scalerValue of 1 as the setting for maximum responsiveness. Its endpoint guidance also suggests setting maximum workers about 20% above expected peak concurrency. Treat these as starting settings derived from your expected traffic rather than fixed deployment requirements.

Record the endpoint identifiers

The RunPod API request guide uses the endpoint ID in the request URL and a Bearer API key in the authorization header. For the synchronous operation, substitute your values as follows:

https://api.runpod.ai/v2/YOUR_VALUE/runsync
Authorization: Bearer YOUR_VALUE

Keep the endpoint ID distinct from MODEL_NAME. The former identifies the Serverless endpoint; the latter tells the worker which weights to load.

Validate an llm deploy through both API routes

The RunPod vLLM worker’s documented OpenAI API compatibility page defines this client base URL:

https://api.runpod.ai/v2/YOUR_VALUE/openai/v1

Configure the base_url in your OpenAI-compatible client with that exact value, replacing only YOUR_VALUE. Do not append the version path a second time if the client already appends its own route.

The Serverless job interface is separate. According to the RunPod request-operation documentation, /runsync waits for completion, and its result remains available for 1 minute, with 5 minutes as the maximum. /run is asynchronous: retrieve its result through /status, where it remains available for 30 minutes after completion.

Rate limits apply per endpoint and operation. For example, /run permits 1000 requests per 10 seconds and 200 concurrent requests; excess requests receive HTTP 429. Follow the current request format for the operation and worker you selected, then replace the endpoint and API-key placeholders.

Treat deployment as verified only after the intended interface returns a usable result. For the OpenAI-compatible interface, confirm that your client can submit a request through the configured base URL. For Serverless jobs, retrieve the /runsync result or poll /status for /run before the documented availability window ends. Creating the endpoint alone does not verify either path.

Calculate Serverless billing before approval

The RunPod Serverless pricing documentation states that billing begins when a worker starts and continues until it fully stops, rounded up to the nearest second. Its billable phases include worker start time, execution time, and the idle-timeout period. Start time includes container initialization and model loading.

Fill this checklist from the provider’s current pricing page and your own quote or billing information:

The current per-second rate for each selected worker configuration.
The number of workers expected to run.
Each worker’s expected start-to-stop duration.
The configured idle timeout.
Whether any workers will remain active during idle periods.

Use this calculation for each worker, then add the results:

BILLED_SECONDS = CEILING(START_TO_STOP_SECONDS)

ESTIMATED_TOTAL = SUM(
  BILLED_SECONDS_FOR_EACH_WORKER
  * CURRENT_PER_SECOND_RATE_FOR_EACH_WORKER
)

The start-to-stop duration must include initialization, execution, and any billable idle period. Use the provider’s current rate rather than freezing a price from an older quote, and retain the assumptions beside the result so you can recalculate when worker behavior or traffic changes.

Common failure points

The wrong model is loaded: Recheck MODEL_NAME. The vLLM environment-variable reference requires a Hugging Face repository ID or local filesystem path, while MAX_CONCURRENCY is a RunPod-side worker limit rather than a vLLM engine argument.
The client calls the wrong interface: Decide whether the application uses the OpenAI-compatible base URL, /runsync, or /run. These paths serve different client flows and should not be substituted for one another.
A job stops at the configured boundary: Compare the failure time with both executionTimeout and ttl, then account separately for queue time and execution time.
A result is missing: Retrieve /runsync output immediately, or poll /status for an asynchronous /run job before its documented availability window closes.
Requests receive HTTP 429: Check the limit for the specific endpoint and operation rather than changing model settings; request frequency and endpoint concurrency limits are separate controls.
The bill is higher than expected: Review idle time and worker lifetime. RunPod’s Serverless pricing documentation includes startup and idle-timeout periods in billable worker time, so a quiet endpoint is not automatically free.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.