AI DeploySelf-hosted LLMGet it deployed

Build a Long-Running LLM Inference Stability Test

Set up staged LLM serving tests with documented RunPod limits, Prometheus checks, clear pass criteria, and reusable failure logs.

For llm hosting, build a long-running inference stability test as a controlled soak: freeze the model and runtime settings, drive a fixed request mix through known stages, measure errors and latency, and decide pass or fail against limits recorded before the run. The procedure below uses a RunPod Serverless vLLM endpoint and Prometheus as the reference setup, while remaining adaptable to infrastructure you operate. Its intended output is a reproducible run record rather than a claimed benchmark.

What to prepare for a repeatable llm hosting stability test

Create one immutable run sheet before sending traffic. Record the run ID, UTC start and end, endpoint ID, model ID, image or code revision, worker settings, client version, request-corpus version, and monitoring configuration. Freeze everything except the variable that a particular stage is intended to change.

Choose representative prompts and expected response checks, but keep private payloads out of shared logs; store a corpus version or hash instead. Assign every submitted request a final state: completed, failed, cancelled, or pending. Decide whether retries belong to the workload. If they do, retain the original attempt and every retry as separate records.

Set the thresholds before collecting data and include units beside every time-based value:

MIN_COMPLETION_RATIO=YOUR_VALUE
MAX_ERROR_RATIO=YOUR_VALUE
MAX_P90_LATENCY=YOUR_VALUE
MAX_LONGEST_SUCCESS_GAP=YOUR_VALUE
MAX_REVIVAL_PROBES=YOUR_VALUE
MAX_BILLABLE_WORKER_SECONDS=YOUR_VALUE

Calculate completed / submitted and failed / submitted over the same stage window. Calculate the longest interval between consecutive successful completions with max(success_time[n] - success_time[n-1]). A stage with no submitted requests cannot pass.

Keep the soak duration separate from one job's lifetime

RunPod's endpoint settings list defaults of 0 active workers, 3 maximum workers, a 5 s idle timeout, a 600 s execution timeout, a 24 h job TTL, and FlashBoot enabled.

The same settings start the TTL at submission rather than execution. They also state that exceeding the execution timeout fails the job and stops the worker, while FlashBoot retains worker state after spin-down.

A multi-hour soak should therefore be implemented as a sequence of requests with stage deadlines, not as one request expected to run indefinitely. Record submission, queue, execution, and completion separately so that time spent waiting cannot be mistaken for inference time. Keep the effective timeout and TTL values in the run sheet because documented defaults can change.

Put a billing stop condition around the run

RunPod's Serverless pricing bills from worker start until the worker fully stops, rounded up to the next second. Its documented billable phases cover container initialization and model loading, request execution, and the idle-timeout period.

For each worker lifecycle, calculate:

BILLABLE_SECONDS = ceil(worker_stop - worker_start)
TOTAL_BILLABLE_SECONDS = sum(BILLABLE_SECONDS for every worker)
WORKER_CHARGE = TOTAL_BILLABLE_SECONDS × rate_from_your_current_quote

Read the applicable rate from your own current quote. Pause traffic before the next stage if the accumulated worker time reaches MAX_BILLABLE_WORKER_SECONDS. Client request duration is not a substitute for worker lifetime: initialization and idle periods must remain visible in the accounting.

Drive controlled traffic through the endpoint

RunPod's request documentation uses endpoint-specific URLs with a Bearer API key. Its /run operation processes jobs in the background; results are retrieved through /status and remain available for 30 minutes after completion.

The documented request target has this form:

POST https://api.runpod.ai/v2/ENDPOINT_ID/run
Authorization: Bearer YOUR_VALUE

Use the request body documented for your deployed worker rather than inventing fields that the test does not need. The same request documentation lists the /run limit as 1,000 requests per 10 seconds and 200 concurrent requests per endpoint and operation; excess requests receive HTTP 429.

Run the traffic plan in this order:

  • Preflight: Send one valid request, retrieve its result, and verify the expected response shape.
  • Steady stage: Hold the selected request mix and concurrency for STEADY_DURATION=YOUR_VALUE.
  • Load steps: Increase one concurrency level at a time and hold it for HOLD_AT_EACH_LEVEL=YOUR_VALUE.
  • Idle and revival: Stop traffic for IDLE_PROBE_AFTER=YOUR_VALUE, send a revival probe, and measure the first successful completion.
  • Recovery: Return to the baseline request mix and require the completion, error, and latency gates to recover.

For every attempt, record the send timestamp, first-response timestamp, completion timestamp, final status, attempt number, and validation result. These fields let you distinguish a pause between successful requests from a change in the average response duration.

Pin the vLLM settings that affect memory

The vLLM memory guidance makes several controls relevant to a stability run. tensor_parallel_size splits a model across GPUs, but each process reads the whole checkpoint, increasing disk-loading time as tensor parallelism grows. Devices should be selected with CUDA_VISIBLE_DEVICES rather than with torch device calls before vLLM starts. Lowering max_model_len or max_num_seqs can reduce memory use, while CUDA graphs are enabled by default and consume additional GPU memory; enforce_eager disables graph capture entirely.

Record the effective values alongside the run:

CUDA_VISIBLE_DEVICES=YOUR_VALUE
tensor_parallel_size=YOUR_VALUE
max_model_len=YOUR_VALUE
max_num_seqs=YOUR_VALUE

Do not change these values halfway through a stage. Changes to context length, batch limits, device placement, tensor parallelism, or graph capture alter the workload or memory conditions being evaluated. If a change is necessary, start a separately labelled stage and do not combine its results with the preceding configuration.

Configure Prometheus and calculate P90 correctly

The Prometheus getting-started configuration places scrape targets under scrape_configs; its example uses a 15s global scrape interval. Make that interval an explicit test setting rather than assuming it applies to every deployment:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'YOUR_VALUE'
    static_configs:
      - targets:
          - 'YOUR_VALUE'

Start Prometheus with the documented command:

./prometheus --config.file=prometheus.yml

Archive the configuration with the run sheet. If the service exposes a classic request-duration histogram named http_request_duration_seconds_bucket, the Prometheus query-function documentation gives this P90 query:

histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))

For a classic histogram, retain le in the by clause. Pair the P90 result with request-log calculations for completion ratio, error ratio, and longest success gap. Save the exact query text because changing aggregation labels can produce a different result.

Apply stage and overall pass criteria

Evaluate each stage against the same definitions:

Signal Calculation Pass rule
Submitted requests submitted > 0
Completion ratio completed / submitted >= MIN_COMPLETION_RATIO
Error ratio failed / submitted <= MAX_ERROR_RATIO
Tail latency P90 query result <= MAX_P90_LATENCY
Longest success gap max(success_time[n] - success_time[n-1]) <= MAX_LONGEST_SUCCESS_GAP
Revival probes count(revival_probes) <= MAX_REVIVAL_PROBES
Worker exposure TOTAL_BILLABLE_SECONDS <= MAX_BILLABLE_WORKER_SECONDS

At every stage boundary, export the counters and calculate all gates from the same window. Mark the overall run as passed only when every mandatory stage passes. A missing metric, missing log field, or unclassified request makes the test result inconclusive rather than successful.

Record failures before retrying

Write a failure row before any retry. Include the run ID, stage, request ID, timestamps, HTTP or transport status, error text, model and configuration revision, attempt number, and any worker lifecycle fields exposed by the platform. Store input and output hashes when the original payloads cannot be retained.

The most common distorted results come from allowing retries to overwrite the original error, using averages instead of tail latency and success gaps, or changing runtime settings during a stage. Keep timeout, cancellation, rejected request, worker stop, malformed output, and partial output as distinct categories. At the end, reconcile the request total:

submitted = completed + failed + cancelled + pending

Assign each request exactly one final state. An unexplained row is a failed test audit even if the service otherwise appears healthy. Treat result-retention checks and billing limits as pass/fail inputs, not post-run cleanup.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources