Build a Repeatable LLM Inference Benchmark

A repeatable LLM inference benchmark is a controlled experiment: freeze the model, dataset, serving configuration, request pattern, and metric definitions, then repeat the same run without changing them. For a self hosted llm, drive a running endpoint with a documented workload, retain raw results and the run manifest, and change only the variable under test. Compare results only when the timing boundaries and workload settings match.

Prepare a fixed system under test

If you intend to report an MLPerf result, the MLPerf Inference Rules require every submission to use LoadGen, provided in C++ with Python bindings. LoadGen generates queries for each scenario, tracks latency, validates accuracy, and computes final metrics. The rules define latency from the moment LoadGen schedules a query for the system under test until it receives a reply.

The same rules distinguish these load patterns:

Scenario	Documented arrival behavior	Documented result
Server	Queries follow a Poisson distribution	Maximum Poisson throughput supported under a benchmark-specific latency constraint
Offline	All samples are sent at the start in a single query	Measured throughput, with no latency constraint
Single stream	The next query starts after the previous one completes	A 90th-percentile early-stopping latency estimate

For LLM benchmarks, MLPerf collects time to first token (TTFT) and time per output token (TPOT). Server latency constraints are expressed as category-specific TTFT/TPOT pairs. The system under test must also be a defined set of hardware and software resources, and it should not retain information associated with query indices.

Whether or not you use MLPerf, create a run manifest before sending traffic. It should contain:

• System: GPU model, CPU, memory, driver and runtime identifiers, inference-server version, server flags, model revision, quantization choice, endpoint, and streaming mode.
• Workload: dataset revision, request rate, concurrency limit, traffic distribution, ramp strategy, and generation settings sent by the client.
• Execution: exact benchmark command, client version, run order, server lifecycle, relevant cache state, raw report, and any client errors.

For components that use semantic versioning, record exact versions rather than moving labels. Semantic Versioning 2.0.0 says a released version must not be modified, and it defines MAJOR for incompatible API changes, MINOR for backward-compatible functionality, and PATCH for backward-compatible fixes. That makes a version change an explicit benchmark variable rather than an invisible system change.

For file-based benchmark data, the DVC data-versioning guide documents that dvc add records a small .dvc metadata file for Git, the data hash determines its cache path, and dvc push uploads cached data to a configured remote. Its restoration workflow checks out the recorded .dvc revision with Git and then runs dvc checkout.

dvc add YOUR_VALUE
dvc push

Store the .dvc metadata revision with the run manifest. That lets another operator identify the intended data even when the data itself remains in a separate cache or remote.

Run the self hosted llm benchmark

Use the vLLM Benchmark CLI documentation as the contract for the following workflow. It positions vllm bench for feature evaluation and regression testing and recommends GuideLLM for production vLLM servers.

For the documented ShareGPT workflow, download the dataset with the published command:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

With the model server already running, the documented online benchmark command is:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

Keep the model identifier identical to the model exposed by the server. The command uses the documented /v1/completions endpoint path; the server-start configuration remains a separate prerequisite and should come from the current vLLM serving documentation.

ShareGPT is documented for online and offline benchmarks. If you use a private corpus instead, the documented contract is a .jsonl file with a prompt field in each entry, selected with --dataset-name custom. Check the current CLI page for the file-location option rather than inventing an undocumented flag.

For a controlled-load variant, append the documented load flags to the command:

--request-rate YOUR_VALUE \
--max-concurrency YOUR_VALUE \
--burstiness YOUR_VALUE \
--ramp-up-strategy YOUR_VALUE

--request-rate controls requests per second. Its documented default is inf, which sends all requests immediately for maximum-throughput testing; a finite value creates controlled load. --max-concurrency caps outstanding requests, while omitting it leaves concurrency unlimited. --burstiness controls variability through a Gamma distribution, with 1.0 representing Poisson traffic. --ramp-up-strategy accepts linear or exponential. Resolve these placeholders before the run and preserve the values in the manifest.

Verify the metrics and preserve the evidence

Do not treat a throughput value without its timing definition as a complete result. The vLLM report lists request throughput, output-token throughput, and total-token throughput, together with the following latency fields:

Metric	Documented measurement boundary
TTFT	Client-side time from sending the request to receiving its first streamed output
ITL	Time between consecutive streamed outputs, aggregated over successful requests
TPOT	Calculated once per request, excluding the first token, and then aggregated across requests
Throughput	Request throughput in requests per second, plus output-token and total-token throughput

With speculative decoding, one streamed output can contain several tokens. The vLLM documentation explains that ITL records gaps only between streamed outputs and does not insert zero-duration gaps for tokens delivered in the same output, so ITL and TPOT can diverge. Preserve that distinction instead of renaming one metric to match another.

A run is ready for comparison when the client finishes without a reported error, the report contains the expected throughput and latency fields, and the saved manifest identifies the same model, dataset revision, endpoint, server configuration, and load profile. Archive the raw report with the command and manifest. Then repeat the identical procedure and inspect variation across runs; a single result is an observation, not evidence of repeatability.

Where the benchmark usually becomes misleading

• Cache-assisted reruns: Repeating vllm bench serve against the same server can reuse prompts left in the prefix cache and inflate throughput. Record server lifecycle and run order, and do not describe an uncontrolled warm rerun as an independent baseline.
• A moving system: A model revision, server flag, runtime, hardware change, or client update can alter the system under test. Record each change and rerun the complete comparison rather than combining older and newer results.
• Different arrival patterns: A maximum-throughput run with --request-rate inf is not comparable to a finite-rate run. Concurrency, burstiness, and ramp strategy are workload variables, not interchangeable report labels.
• Metric-label confusion: Benchmarking tools do not standardize metric terminology. Compare TTFT, ITL, and TPOT only after confirming where each timer starts and stops.
• A moving client boundary: TTFT is measured by the client. If the endpoint is remote, the network path is part of that observation, so keep client location and connection path stable.

Another operator should be able to use the saved manifest and raw outputs to reconstruct the same system, dataset, request pattern, and measurement interpretation. If any of those elements is missing, report the result as a diagnostic observation rather than a performance comparison.

These steps are compiled from the official documentation linked in the text, nothing was run or measured for this article, and the commands and definitions should be checked against the current version before use.

Sources

• MLPerf Inference Rules
• vLLM Benchmark CLI
• Semantic Versioning 2.0.0
• DVC Data Versioning