Run Non-Real-Time LLM Jobs on RunPod Spot
Learn how interruptible RunPod Spot Pods can stop at any time, then run resumable LLM jobs with persistent checkpoints and controlled recovery.
For ai model deployment involving batch LLM work, use a RunPod Spot Pod only when the application can tolerate a stop and resume from durable state. The RunPod Create Pod API reference defines this mode with the interruptible boolean: set it to true, and the Pod can be stopped at any time. Build checkpointing and resubmission into your own workflow rather than assuming the Pod restarts or replays the job.
A resumable design makes ai model deployment safer on Spot
Treat interruption as a normal execution branch rather than an exceptional failure. Before launching a batch job, confirm that:
- Its completion deadline can move without breaking a user-facing commitment.
- Re-running a partially processed item will not duplicate harmful side effects.
- Each work unit can produce a complete checkpoint independently.
- Inputs, configuration, and outputs can be restored without relying on process memory.
- The application can reject or replace an incomplete checkpoint safely.
A job is not non-real-time merely because it uses a batch script. If a person is waiting for an immediate response, a single run cannot tolerate repetition, or progress exists only in memory, Spot interruption is a poor fit.
Put resumable state on the right storage
RunPod’s storage options document that container storage is temporary and cleared when a Pod stops, data under /workspace survives stops and restarts but is deleted when the Pod is terminated, and a network volume exists independently and remains after the Pod is deleted.
For checkpoints, manifests, and recoverable partial output, use /workspace rather than the container disk. Write new state without replacing the previous complete checkpoint until the new state is ready. This lets the next process reject an incomplete write and continue from the last known-good point.
Use a network volume when outputs must outlive Pod termination or deletion. That retention does not remove the need for a backup: RunPod’s backup guidance says its storage is not intended for long-term cloud storage and recommends copying critical data to a local machine or dedicated cloud storage provider.
Calculate the cost before allowing retries
RunPod’s Pod pricing documentation states that compute and container or volume disks are billed per second, network volumes are billed hourly, and data ingress and egress carry no charge. Storage charges can continue while a Pod is stopped, so the cheapest execution attempt may still leave storage charges behind.
Copy the current rates from the provider’s page and fill in your own values rather than relying on a cached figure:
YOUR_HEADROOM =
YOUR_ACCOUNT_BALANCE - YOUR_REQUIRED_CREDIT_RESERVE
YOUR_TOTAL_JOB_COST =
YOUR_GPU_COMPUTE_ACROSS_ALL_ATTEMPTS
+ YOUR_CONTAINER_DISK_CHARGES
+ YOUR_VOLUME_DISK_CHARGES
+ YOUR_NETWORK_VOLUME_CHARGES
Include failed attempts and storage retained after interruption in the calculation. Published rates and account conditions can change, so calculate again before deployment.
Size the GPU before tuning vLLM
RunPod’s Pod selection guide gives approximately 2 GB of VRAM per billion parameters as a rule of thumb. Use that figure to shortlist GPU memory, not as proof that a particular model, context length, and concurrency setting will fit.
The vLLM serve CLI reference documents the memory fraction, context-length, sequence, and KV-cache controls relevant to this workload. Add the appropriate arguments to the model-loading command required by your selected model project:
--gpu-memory-utilization 0.92
--max-model-len auto
--max-num-seqs YOUR_VALUE
--gpu-memory-utilization controls the fraction assigned to the model executor; its documented default is 0.92, and the limit applies per vLLM instance. --max-model-len covers prompt plus output: when omitted, vLLM derives it from the model configuration, while auto or -1 selects the largest length that fits in GPU memory. --max-num-seqs caps sequences processed in an iteration.
If you set a direct KV-cache allocation, use the documented argument:
--kv-cache-memory-bytes YOUR_VALUE
The same vLLM reference states that this setting overrides --gpu-memory-utilization. Choose either the direct KV-cache allocation or the memory-fraction approach deliberately; do not assume both independently control the same memory budget.
Create the Spot Pod through the documented API
The Create Pod reference gives interruptible a default of false, so do not omit the field when Spot behavior is required. The documented request method, endpoint, and authorization header are:
curl --request POST \
--url https://rest.runpod.io/v1/pods \
--header 'Authorization: Bearer RUNPOD_API_KEY'
That shell fragment is not a complete creation request because the remaining body fields depend on the selected configuration. Build the full body from the current API schema and include this Spot selector:
{
"interruptible": true
}
Do not guess model, GPU, image, storage, or port field names. Likewise, the RunPod API reference does not prescribe a complete LLM model-loading command; obtain that part from the selected model project and the matching vLLM documentation.
Verify application recovery, not merely Pod creation
Use a known input fixture and validate the resulting artifact against an expected condition. A usable verification sequence is:
- Confirm that the model process reaches the ready state defined by its own runtime.
- Run a fixture that should produce a recognizable, deterministic output.
- Check that the job records completion before starting dependent work.
- Confirm that the latest complete checkpoint is readable from the selected storage location.
- Exercise application restart logic and verify that it resumes from that checkpoint rather than an uncommitted object.
Run these checks before allowing unattended retries. Record the expected next work item, its input location, and the validation rule; an exit without one of these values is not a recoverable result.
Handle expected failures without losing the job
A restart can find zero GPUs
RunPod’s zero-GPU troubleshooting page explains that a stopped Pod remains tied to its physical machine while its released GPU may be rented by someone else. A later restart can therefore find no GPU available.
Do not interpret this event as proof that /workspace was deleted. The troubleshooting page does not document automatic migration of local storage to another machine, so keep recovery instructions aligned with the storage actually attached to the Pod.
A zero balance can stop running Pods
RunPod’s Pod pricing documentation states that when the balance reaches $0, running Pods stop; Pods with a network volume are preserved, while Pods without one are terminated and their data cannot be recovered. This published threshold was checked on 2026-10-01 and may change.
Set an internal balance reserve based on your longest expected attempt and storage-retention period. A network volume can preserve a Pod at the zero-balance boundary, but it does not justify treating the balance as irrelevant.
Retries need a ceiling
Cap retries in your own scheduler and include completed attempts in YOUR_TOTAL_JOB_COST. Increase the cap only when evidence shows that another full attempt is likely to complete useful work. Avoid automatic create-delete loops that repeatedly rebuild state already available under /workspace.
Final operating checklist
Before release, verify that the workload has deadline slack, idempotent work units, complete checkpoints, an explicit storage policy, current vLLM arguments, a complete API request body, a balance reserve, and a bounded retry policy. After an interruption, restore the last complete checkpoint, validate it, and continue from the recorded next item rather than restarting an already completed unit.
These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.