AI DeploySelf-hosted LLMGet it deployed

Test RunPod Serverless Concurrency for LLM Apps

Test RunPod Serverless concurrency with controlled bursts, documented limits, TTFT, inter-token latency, worker settings, and billing checks.

Test RunPod Serverless concurrency with controlled request bursts while recording request outcomes, time to first token and time per output token, completion throughput, and whether workers were already warm. Compare every run with the same payload set and change one endpoint or engine setting at a time. For an llm deploy, the useful result is the highest tested concurrency that still meets your own correctness and latency criteria—not an unverified maximum.

What an llm deploy concurrency test must measure

Before sending traffic, freeze a run manifest containing the endpoint ID, worker image, model, endpoint controls, engine arguments, request corpus, and validation rule. If one of these changes, the result describes a different system.

Record the following for every request:

  • Correctness: submission result, terminal status, returned output, and whether streaming ended cleanly.
  • Load: requests submitted, requests in flight, successful completions, failures, and rate-limit responses.
  • Timing: request submission, first token, intermediate tokens, final token, and terminal response.
  • Scaling: initial worker condition, maximum-worker setting, scaler mode, queue observations, and traffic pattern.
  • Billing: test-phase boundaries, idle settings, and the provider-reported billing record.

Keep observations taken during worker startup separate from those taken after workers are warm. Do not combine a short burst with a steady-load run, or streaming requests with non-streaming requests. Select the percentile and aggregation method before reviewing the results, and retain the raw per-request rows so an aggregate can be checked later.

Prepare the endpoint and vLLM settings

RunPod’s Endpoint settings list defaults of 0 active workers, 3 maximum workers, a 5 s idle timeout, a 600 s execution timeout, a 24 h job TTL, and enabled FlashBoot. Record these values rather than assuming they remain unchanged.

RunPod’s vLLM environment-variable reference says MODEL_NAME accepts a Hugging Face repository ID or local filesystem path and defaults to facebook/opt-125m. It also defines MAX_CONCURRENCY as a Runpod-side limit on concurrent requests per worker, with a default of 30; it is not a vLLM engine argument.

Use the deployment mechanism appropriate to your worker image to replace these placeholders:

MODEL_NAME=YOUR_VALUE
MAX_CONCURRENCY=YOUR_VALUE

Keep the model, MAX_CONCURRENCY, maximum workers, request payload, generation settings, and stream mode fixed during the baseline. This makes a later change attributable to one setting rather than to an undocumented configuration mix.

Establish the documented operating limits

RunPod’s Send API requests documentation defines rate limits per endpoint and operation. For /run, it lists 1000 requests per 10 seconds and 200 concurrent requests; requests over the applicable limit receive HTTP 429.

That allowance answers how quickly your client may submit requests. It does not establish how many LLM requests the endpoint can complete within your latency target, because worker startup, model execution, request size, and scaler behavior still affect the result. Do not treat the operation limit as an application-capacity result.

Define the stopping rule before the run. It should identify the first condition that ends a concurrency level, such as a correctness failure, an unacceptable timing value, a missing response, or excessive resource use. Continuing after the first failure can hide where the system stopped meeting the requirement.

Drive requests and record timing

The RunPod request documentation uses a Bearer API key with the endpoint forms shown below:

https://api.runpod.ai/v2/ENDPOINT_ID/run
https://api.runpod.ai/v2/ENDPOINT_ID/runsync
https://api.runpod.ai/v2/ENDPOINT_ID/status
Authorization: Bearer YOUR_VALUE

Use the payload schema accepted by your deployed handler rather than copying a request body from another API. During each level, record the initial worker condition, release the prepared request set, capture timestamps and outcomes, and wait for every submitted request to reach a terminal state.

for each configured concurrency level:
    record the initial worker condition
    release the prepared request set
    record request_sent, first_token, intermediate_tokens, final_token, and outcome
    wait for all submitted requests to finish or fail
    save the results before changing another setting

Apply the linked LLM metric definitions with per-request timestamps:

TTFT_i = first_token_time_i - request_sent_time_i
token_interval_i,j = token_time_i,j - token_time_i,j-1
TPOT_i = mean(token_interval_i,j)
submission_rate = submitted_requests / observation_seconds
completion_rate = successful_completions / observation_seconds
failure_ratio = failed_requests / submitted_requests

Store the formulas and observation window beside the results. Otherwise, throughput values from different phases can be compared even though they do not describe the same traffic pattern.

Change scaling controls separately

RunPod’s queue-delay setting adds workers when requests exceed a waiting threshold, which defaults to 4 seconds. This behavior matters when requests spend time queued before reaching a worker.

For LLM workloads, the page’s request-count scaling guidance documents this target:

Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)

The same guidance suggests setting maximum workers about 20% above expected peak concurrency. Treat that as configuration guidance, not as a measured capacity result.

First hold the request generator and endpoint limits constant while changing offered load. Then return to the baseline manifest, hold the traffic pattern constant, and change the scaler. Finally, test the maximum-worker setting separately. Resetting the manifest between stages prevents a scaler change from being confused with a worker-limit change.

Separate RunPod and vLLM concurrency controls

The vLLM environment-variable table places MAX_CONCURRENCY on the RunPod side. Record it separately from the engine controls below; changing one does not demonstrate the same effect as changing the other.

The project’s vLLM serve CLI reference documents these engine arguments:

Argument Documented behavior
--gpu-memory-utilization Sets the GPU-memory fraction for the model executor; the documented default is 0.92.
--max-model-len Covers prompt plus output context; when omitted, vLLM derives it from the model configuration.
--max-num-seqs Caps the number of sequences processed in one iteration.

A documented argument pattern is:

vllm serve YOUR_VALUE \
  --gpu-memory-utilization YOUR_VALUE \
  --max-model-len YOUR_VALUE \
  --max-num-seqs YOUR_VALUE

This is a direct vLLM server pattern, not a RunPod endpoint request. Verify separately whether your deployed worker image accepts these arguments.

The same CLI reference says --kv-cache-memory-bytes sets KV-cache size per GPU and, when provided, overrides --gpu-memory-utilization. Do not change both controls during an unexplained run, and do not attribute a scaling result to one while the other also differs.

Account for billing during the test

RunPod’s Serverless pricing documentation says billing starts when a worker starts and continues until it fully stops, rounded up to the next second. It divides billable worker time into start time, execution time, and the idle-timeout period; it describes start time as container initialization and model loading, with the idle timeout defaulting to 5 seconds.

Copy the provider’s usage record for each phase instead of reconstructing the bill from client response times. At minimum, retain the worker-start and full-stop boundaries, idle setting, active-worker setting, and provider-reported duration. Compare concurrency runs only when those billing fields are aligned, otherwise a change in worker lifetime can obscure the effect of request load.

Check that the run worked

A concurrency run is usable only when its evidence supports the comparison:

  • Every submitted request has a recorded terminal outcome.
  • Successful responses pass the application’s output validator.
  • Streaming responses end cleanly when streaming is required.
  • TTFT and token intervals come from the original timestamps.
  • The endpoint, model, scaler, and engine settings match the run manifest.
  • Rate-limit responses and failed requests remain included in the results.
  • Worker settings and provider billing data are attached to the test.

Report the highest concurrency level that passed every criterion, together with the raw request count, timing metrics, failures, and configuration. If a run fails, distinguish a capacity boundary from an invalid test. Do not extrapolate beyond the highest level actually exercised under the recorded conditions.

Where concurrency tests usually go wrong

  • Mixing worker states: startup observations and warm-worker observations answer different questions.
  • Changing two controls together: a new scaler and a new worker limit make attribution unreliable.
  • Confusing limits with capacity: an accepted request is not necessarily a timely, correct completion.
  • Conflating control layers: MAX_CONCURRENCY, maximum workers, --max-num-seqs, and memory settings belong to different parts of the deployment.
  • Ignoring override behavior: resolve the documented KV-cache override before comparing memory configurations.
  • Comparing unlike requests: changes in payload, generation settings, or streaming behavior invalidate a direct comparison.
  • Rebuilding billing from client timing: provider usage records are the billing evidence, not request latency.

These steps are compiled from the official documentation linked in the text, and no endpoint was exercised for this article; check the current versions before use.

Sources