Benchmark a vLLM Deployment with Replayed Traffic

Replay a representative JSONL prompt set through vLLM and compare TTFT, ITL, TPOT, and throughput under controlled request rates and concurrency.

A repeatable llm deploy benchmark with replayed traffic should drive an already running vLLM server with a saved, representative prompt set under a declared arrival pattern, then keep the dataset, load settings, server configuration, and raw results together as one run record. Keep expected-demand replay separate from maximum-capacity testing: the first checks behavior under a chosen load, while the second asks how much the deployment can push.

What makes an llm deploy replay representative?

A useful replay preserves workload shape rather than repeating a convenient prompt list. Build it from sanitized captures or a representative corpus, retaining routine requests, long inputs, and boundary cases. Do not place credentials or private customer content in the file. If results also depend on generation settings, endpoint type, or arrival timing, preserve those values separately; a prompt file alone does not prove that every captured request attribute was replayed.

Write the run contract before execution:

Question: expected demand, maximum capacity, regression detection, or endpoint overhead.
Workload: immutable prompt set, selection rules, and file identity.
Deployment: model, server command, runtime, host, GPU allocation, and container image.
Load: request rate, concurrency, burst shape, ramp policy, and cache state.
Output: raw report location and the conditions needed to reproduce the run.

Marking these conditions prevents a changed server flag, warmed prompt set, or different traffic shape from being presented as a deployment regression.

Confirm the endpoint and benchmark contract

The vLLM OpenAI-Compatible Server documentation says vLLM's HTTP server implements OpenAI's Completions and Chat APIs. The /v1/completions endpoint applies to text generation models and does not support suffix; /v1/chat/completions requires a chat template and ignores the user parameter. Replay the API family your deployment actually serves rather than assuming that compatible endpoint names produce identical request behavior.

The vLLM Benchmark CLI documentation defines the benchmark contract used here:

Dataset: A custom dataset uses JSONL with a prompt field per entry and is selected with --dataset-name custom. The documentation uses data.jsonl as the filename example. ShareGPT is also supported for online and offline benchmarks.
Online runner: vllm bench serve points at a running server. Its documented example supplies the vllm backend, a model, and an endpoint such as /v1/completions.
Results: The report contains request throughput, output-token throughput, and total-token throughput. TTFT is client time from request submission to the first streamed output; ITL records the gap between consecutive streamed outputs; TPOT is calculated once per request excluding the first token. With speculative decoding, one streamed output can contain several tokens, so ITL and TPOT can diverge.
Load: --request-rate sets the target requests per second; finite values provide controlled simulation, while inf requests maximum throughput and is the default that sends all requests immediately. --max-concurrency caps outstanding requests and is unlimited when omitted. --burstiness shapes arrivals with a Gamma distribution, while --ramp-up-strategy accepts linear or exponential.
Comparability: Repeating the benchmark against the same server can reuse prompts in its prefix cache and inflate throughput. Metric terminology is not standardized across benchmarking tools.
Scope: The documentation positions vllm bench for feature evaluation and regression testing and recommends GuideLLM for production-server benchmarking.

Create the custom file with the documented field:

{"prompt":"YOUR_VALUE"}

If your current CLI requires a separate custom-file path control, take its exact name from the current vllm bench serve reference. The --dataset-name custom value selects the dataset mode; it is not a filesystem path.

Run the replay with an explicit load profile

After adding the custom-file control shown by the current CLI reference, run the documented portion of the command:

vllm bench serve \
  --backend vllm \
  --model YOUR_VALUE \
  --endpoint /v1/completions \
  --dataset-name custom \
  --request-rate YOUR_VALUE \
  --max-concurrency YOUR_VALUE \
  --burstiness YOUR_VALUE \
  --ramp-up-strategy YOUR_VALUE

Fill YOUR_VALUE rather than accepting settings accidentally. Use the exact model served by the deployment, choose a finite request rate for expected-demand replay, and set concurrency deliberately instead of leaving it implicit. Select the ramp policy from the documented linear or exponential values.

Do not combine a finite-rate replay with an inf run merely to increase the sample size. They represent different load questions. Likewise, do not change the model, endpoint, dataset, and traffic shape together: the resulting comparison will not identify which change caused the difference.

Check what the report proves

Verification should establish that the intended workload ran, rather than merely confirming that the command finished.

Run identity: Match the report to the saved prompt file, model, endpoint, request rate, concurrency, burst setting, and ramp policy.
Metric consistency: Retain TTFT, ITL, TPOT, and each throughput field under the definitions in the benchmark contract. Do not compare labels without checking their meanings.
Cache declaration: Record whether the run started with prompts already present on the server. Keep warm and cold scenarios visibly separate.
Failure visibility: Preserve the raw benchmark output and the server logs covering the same run window. A partial report should not be compared as if it represented the whole replay.
Attribution: Keep direct vLLM and gateway measurements separate when you need to locate delay.

If a custom FastAPI gateway sits in the measured path, include it. FastAPI's streaming guidance says the yield-based StreamingResponse style handles cancellation, while the application remains responsible for producing and encoding the streamed bytes. Benchmarking the direct vLLM route and the complete gateway route answers different operational questions.

Where replay benchmarks usually go wrong

Most misleading results come from changes in the run contract:

Cache carry-over: Mark warm and cold runs separately; do not silently combine repeated identical prompt sets.
Accidental saturation: Keep finite-rate and maximum-throughput profiles in separate result groups.
Concurrency drift: Record the cap used for every run, including policies that intentionally leave requests uncapped.
API drift: Keep the endpoint, model type, and request fields aligned with the deployment being evaluated.
Metric drift: Preserve metric definitions and units instead of matching results by field name alone.
Configuration drift: Do not attribute a change to the model or traffic when the server command, runtime, container, or cache state also changed.

These controls make a failed run diagnosable instead of forcing you to guess which variable moved.

Decide whether the run is an MLPerf result

The MLPerf Inference Rules require every submission to use LoadGen, which generates queries for each scenario, tracks latency, validates accuracy, and calculates the final metrics. In the Server scenario, LoadGen uses Poisson arrivals and reports the maximum Poisson throughput supported under a benchmark-specific latency constraint. Offline sends all samples at the start, while Single stream sends the next query only after the previous one completes.

The rules define the system under test as a defined set of hardware and software resources and state that it should not cache information associated with query indices. A custom vLLM JSONL replay can support internal deployment decisions, but it does not become an MLPerf result merely because it reports similar metrics. Follow the current submission process and rules before making that designation.

Make the run repeatable

Archive the immutable dataset identity, exact server and model configuration, endpoint, custom-file input, load profile, cache state, raw report, and relevant logs. Before accepting a comparison, require the model, dataset, endpoint, and load policy to match. Otherwise, label the result as exploratory rather than treating it as a deployment comparison.

These steps are compiled from the official documentation linked in the text, and nothing was run or measured for this article. Check the commands and metric definitions against the current version before use.