Build a Self-Hosted LLM Performance Test Harness

A reliable self hosted llm performance test harness fixes the model and serving stack, drives a documented API with a repeatable workload, and records latency, throughput, and request cost without mixing definitions. Keep the system under test, load generator, workload, measurements, and deployment context separate so another operator can repeat the run. The structure below comes from official documentation; no benchmark was run for this article.

Preserve the self hosted llm system under test

MLPerf Inference Rules define the system under test as a defined set of hardware and software resources measured for performance. Your run manifest should therefore capture both layers, along with the model and workload.

Create a harness-owned record such as this:

{
  "run_id": "YOUR_VALUE",
  "system_under_test": {
    "hardware": "YOUR_VALUE",
    "software": "YOUR_VALUE",
    "model_artifact": "YOUR_VALUE",
    "endpoint_path": "YOUR_VALUE"
  },
  "workload": {
    "source": "YOUR_VALUE",
    "version": "YOUR_VALUE",
    "load_generator": "YOUR_VALUE",
    "load_generator_version": "YOUR_VALUE"
  },
  "evidence": {
    "raw_result": "YOUR_VALUE",
    "cost_quote_or_bill": "YOUR_VALUE"
  }
}

This JSON is harness metadata, not an API payload or vendor configuration. Keep it beside the workload and untouched result output; if hardware, software, model, or workload changes, record a new run rather than overwriting the previous one.

Decide whether the benchmark is formal or operational

If the result is intended for an MLPerf submission, the same MLPerf Inference Rules require the C++ LoadGen tool with Python bindings for every submission. LoadGen generates scenario queries, tracks latency, validates accuracy, and computes the final metrics.

Choose the scenario before configuring the harness:

• Server: LoadGen sends queries according to a Poisson distribution. The reported result is the maximum Poisson throughput supported under a benchmark-specific latency constraint.
• Offline: LoadGen sends all samples at the start in a single query. There is no latency constraint, and the metric is measured throughput.
• Single stream: LoadGen sends the next query after the previous one completes. The metric is a 90th-percentile early-stopping latency estimate.

For LLM benchmarks, MLPerf records time to first token, or TTFT, and time per output token, or TPOT, the average interval between generated tokens. The system under test should not cache information associated with query indices. An internal test may use a different documented load generator, but its output must not be presented as an MLPerf submission result unless it follows the applicable rules.

Pin the endpoint and request behavior

The vLLM project's OpenAI-Compatible Server documentation states that its HTTP server implements OpenAI's Completions and Chat APIs. Select the endpoint before preparing the workload:

/v1/completions
/v1/chat/completions

The Completions endpoint does not support the suffix parameter. The Chat Completions endpoint requires a chat template and ignores the user parameter. Record the selected path and obtain its current request schema from the linked documentation. Compatibility with an OpenAI-style API does not make an ignored or unsupported field influence the generated workload.

Select the runner without freezing obsolete tooling

NVIDIA's GenAI-Perf README says that new features are no longer being developed and directs new benchmarking work to AIPerf. A new harness should therefore consult AIPerf's current project documentation for installation and backend-specific invocation instead of copying an unverified legacy command.

For an existing GenAI-Perf harness, the documentation provides this exact streaming example:

genai-perf profile \
  -m gpt2 \
  --backend tensorrtllm \
  --streaming

Treat those values as the documented example, not as defaults for your deployment. The --measurement-interval flag sets the sampling window in milliseconds. With the default stability settings, GenAI-Perf benchmarks for 3 × interval; it considers a measurement stable when the max-to-min ratio of the last 3 measurements is within the stability percentage for both throughput and latency.

Preserve each metric's definition

The metric definitions in the GenAI-Perf README use specific denominators that should remain attached to every stored result:

Metric	Documented definition
Time to First Token	Time between sending a request and receiving its first response
Inter Token Latency	Time between intermediate responses for a request, divided by the number of generated tokens in the latter response
Output Token Throughput	Total output tokens divided by benchmark duration
Request Throughput	Number of final responses divided by benchmark duration
Request Latency	Time between sending a request and receiving its final response
Output Token Throughput Per User	Output tokens excluding the first token, divided by each request's generation-phase duration

Store the tool name, version, definition, and unit with each metric. Do not silently rename GenAI-Perf Inter Token Latency as MLPerf TPOT, and do not combine per-user throughput with whole-benchmark throughput.

Execute and verify the harness

Use the same sequence for every comparable run:

• Freeze the manifest, workload, selected endpoint, and streaming or non-streaming mode before generating load.
• Send a smoke request through that endpoint and mode. Keep its response or failure separate from benchmark results.
• Start monitoring before the load generator and record the benchmark window in the manifest.
• Run the documented loader with validated placeholders, then retain its unmodified result output.
• Compare only runs with the same system definition, workload, request mode, and metric denominator.

The Prometheus query documentation provides this exact P90 expression for a classic histogram:

histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))

Use it only when your exporter actually exposes the referenced histogram. When aggregating a classic histogram, keep the le label in the by clause. Prometheus also explains that client-side precomputed quantiles cannot be aggregated across replicas, while histograms can; follow its histogram guidance when your deployment has multiple replicas.

A run is operationally verified when the endpoint check succeeds, raw tool output is retained, metric names match their definitions, and failed requests are recorded explicitly against the chosen denominator.

Calculate request cost from actual billing

Cost is a separate result derived from your provider's current billing page, quote, or invoice. List only line items actually charged for the benchmark window, such as compute, storage, network, monitoring, or other allocated charges. Do not infer a GPU rate or reuse a rate from another provider or billing configuration.

Fill the placeholders from your own records:

BILLABLE_RUN_COST = CHARGED_COMPUTE + CHARGED_STORAGE + CHARGED_NETWORK + OTHER_ALLOCATED_CHARGES

COST_PER_REQUEST = BILLABLE_RUN_COST / REQUESTS_IN_COST_WINDOW

COST_PER_OUTPUT_TOKEN = BILLABLE_RUN_COST / OUTPUT_TOKENS_IN_COST_WINDOW

Define whether failed or cancelled requests remain in the denominator before comparing runs. Use generated output tokens—not requested tokens—for the output-token calculation, and keep the billing window and allocation method unchanged across comparisons.

Keep the evidence versioned

The DVC data-versioning guide explains that dvc add records tracking metadata in a small .dvc placeholder committed to Git, while the file hash determines its cache path. To reproduce an older dataset, check out the relevant .dvc revision with Git and run dvc checkout; dvc push uploads locally cached data to the configured remote.

A harness workflow can use the documented operations as follows:

dvc add YOUR_VALUE
git checkout YOUR_VALUE
dvc checkout
dvc push

Run dvc push only after configuring the intended remote storage.

Where harnesses usually go wrong

• Changing the endpoint, model, hardware, software, or workload while presenting the rows as one trend.
• Treating an internal operational run as a formal benchmark result.
• Assuming ignored or unsupported request fields affect generation; recheck vLLM's endpoint constraints.
• Renaming metric fields or mixing per-user and whole-benchmark denominators.
• Aggregating replica quantiles from a metric type that cannot support that calculation; follow Prometheus's aggregation guidance.
• Changing the cost window or denominator between otherwise comparable runs.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources

• MLPerf Inference Rules
• GenAI-Perf README
• vLLM OpenAI-Compatible Server
• Prometheus Query Functions
• Prometheus Histograms and Summaries
• DVC Data Versioning