AI DeploySelf-hosted LLMGet it deployed

Scheduling Batch LLM Jobs During Off-Peak GPU Hours

Learn how to schedule batch LLM jobs during verified lower-rate windows while accounting for per-second billing, interruptions, storage, and commitments.

Schedule batch LLM jobs by placing each run in a deadline-aware queue with a GPU cost ceiling, durable checkpoints, and an explicit stop-or-terminate policy, then launch it only during a lower-rate window confirmed from the provider’s current quote. For ai model deployment on rented GPUs, “off-peak” is not a universal tariff: the cited documentation does not define one fixed clock schedule across GPU products. Use the workflow below to acquire capacity, verify placement, and limit interruption and storage exposure.

Define Off-Peak GPU Hours From Current Quotes

The cited Amazon EC2 billing and purchasing options page says EC2 On-Demand Instances are billed by the second, but it does not define a provider-wide off-peak clock. Its Savings Plans, Reserved Instances, and Capacity Blocks terms describe one- or three-year usage commitments, commitments tied to a specific instance type and Region, and reservations for GPU clusters. These are purchase or capacity structures, not evidence that a particular hour is automatically cheaper.

The RunPod Pod pricing documentation says Pod compute and storage are billed per second, data ingress and egress are not charged, and on-demand resources cannot be displaced by other users. An on-demand deployment requires credits covering at least one hour of the selected configuration. The page does not define a clock-based off-peak rate, so use a current quote rather than assuming one.

RunPod’s savings-plan terms are a three- or six-month upfront commitment, are non-refundable, and have a fixed expiration date. After a Pod stops, the plan applies to the next deployment using the same GPU type. Treat that as a commitment schedule, not an automatic off-peak discount.

Record the quote inputs before starting the scheduler:

PRODUCT = YOUR_VALUE
GPU_TYPE = YOUR_VALUE
LOCATION = YOUR_VALUE
PURCHASE_OPTION = YOUR_VALUE
QUOTED_RATE = YOUR_VALUE
QUOTED_UNIT = YOUR_VALUE
WINDOW_START = YOUR_VALUE
WINDOW_END = YOUR_VALUE
MAX_ACTIVE_TIME = YOUR_VALUE
COST_CEILING = YOUR_VALUE

Calculate the estimate without inventing a provider rate:

AWS_ESTIMATE = BILLABLE_SECONDS * QUOTED_SECOND_RATE

RUNPOD_COMPUTE_ESTIMATE = ACTIVE_SECONDS * GPU_COMPUTE_RATE

RUNPOD_STORAGE_ESTIMATE =
  BILLED_CONTAINER_DISK_SECONDS * CONTAINER_DISK_RATE
  + BILLED_VOLUME_DISK_SECONDS * VOLUME_DISK_RATE
  + BILLED_NETWORK_VOLUME_HOURS * NETWORK_VOLUME_RATE

OFF_PEAK_SAVING = COMPARABLE_BASELINE_COST - WINDOW_COST

Fill the billable units from your own invoice or quote. Compare only equivalent GPU, location, storage, and purchase options; otherwise, the saving formula is not comparable.

Schedule the Batch LLM Queue

Give every job a scheduler record containing its model image, batch entry point, input location, checkpoint path, output path, deadline, retry limit, cost ceiling, and resource-cleanup policy. Validate model-specific flags against the current documentation for the runtime you use rather than copying an unverified command into the scheduler.

Represent the workflow as explicit states:

QUEUED -> PROVISIONING -> RUNNING -> COMPLETED
                         -> CHECKPOINTED -> RETRYABLE

Make resource creation idempotent: if the scheduler delivers the same queue item again, it must reuse or identify the existing capacity request rather than create another billable worker. A retry should restore the last durable checkpoint, continue from the recorded item, and stop when the deadline or cost ceiling is reached.

Do not mark a job complete merely because a Pod or Kubernetes workload is running. Completion belongs to the application: the expected output and a completion marker must exist in the configured durable location before the scheduler advances the record.

Configure Kubernetes GPU Placement

For ai model deployment on Kubernetes, GPU admission is a resource-scheduling decision before it is an application concern. The Kubernetes GPU scheduling guide says that, after a vendor device plugin is installed, the cluster exposes GPUs as custom schedulable resources such as nvidia.com/gpu. It also says Node Feature Discovery can discover and label GPU-enabled nodes.

The same guide’s GPU resource rules allow GPUs to be specified only under limits, use the limit as the default request, and require request and limit values to match when both are present. A GPU request without a limit is not allowed. Use this fragment and replace the placeholder before applying it:

resources:
  limits:
    nvidia.com/gpu: YOUR_GPU_COUNT

The NVIDIA GPU Operator installation guide says the operator and its managed components are installed into one namespace selected at installation time. It also says the operator deploys NVIDIA drivers in a container by default and directs you to set driver.enabled to false when the nodes already have drivers installed.

Configure the documented Helm repository with:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

That command configures the repository and refreshes its index. Complete the chart, release, namespace, and driver decisions using the current NVIDIA installation guide before starting billable inference work.

Start a RunPod Batch Worker

The RunPod Pod management guide documents console, CLI, and REST Pod creation. It also directs users who see OCI runtime create failed to use Additional filters → CUDA Versions to select compatible machines.

For the console flow:

  1. Select the GPU type.
  2. Name the Pod.
  3. Optionally choose a template.
  4. Set the GPU count.
  5. Click Deploy On-Demand.

The documented CLI creation form is:

runpodctl pod create --name YOUR_POD_NAME --gpu-id YOUR_GPU_ID --image YOUR_IMAGE

For an API-based scheduler, the documented request method, target, and authorization header are:

curl --request POST \
  --url https://rest.runpod.io/v1/pods \
  --header 'Authorization: Bearer YOUR_API_KEY'

The shown lines do not define the request body. Check the current API page for its required fields before sending a complete Pod-creation request; do not invent a payload or assume the header alone is sufficient.

Verify Scheduling and Job Completion

Require four separate acceptance checks:

  • The provisioning record matches the product, GPU, location, purchase option, rate, and time window used for the estimate.
  • The scheduler exposes and accepts the configured GPU resource before starting the batch runner.
  • The application writes its expected output and completion marker to the configured durable location.
  • The provider’s actual billing record uses the expected line items and billing units rather than the worksheet estimate.

Keep capacity state and application state separate. A running Pod confirms that capacity was obtained, not that the LLM task finished; a completed model task does not confirm that temporary storage was released. Reconcile both states before closing the queue item.

Handle Interruption and Cleanup Failures

A stale rate worksheet can turn an apparent off-peak saving into an overestimate. Refresh the quote immediately before provisioning, and make the scheduler stop rather than cross its cost ceiling when the current price no longer matches the approved calculation.

The AWS Spot interruption notice documentation says EC2 issues a warning two minutes before stopping or terminating a Spot Instance, recommends checking every five seconds, and delivers notices on a best-effort basis. The documented IMDSv2 path is:

http://169.254.169.254/latest/meta-data/spot/instance-action

Read that path continuously, checkpoint before an interruption, and make queued work retryable. Because notices are best effort, the handler must not be the only protection against data loss.

For a rejected Kubernetes GPU configuration, return to the linked resource rules: use a limit, or provide equal request and limit values. Do not add a request-only GPU setting or retry the same invalid manifest indefinitely.

Under the RunPod lifecycle guidance, stopping a Pod releases the GPU, preserves data in /workspace, and clears the container disk. The stopped Pod still incurs volume-disk storage charges. Termination stops that stopped-Pod storage charge, but it permanently deletes data not stored on a network volume, so export required artifacts before terminating.

These steps are compiled from the official documentation linked in the text; nothing was run or measured for it, and the commands, paths, billing terms, and resource keys should be checked against the current versions before use.

Sources