AI DeploySelf-hosted LLMGet it deployed

Run a Self-Hosted LLM Performance Benchmark

Run a repeatable vLLM benchmark that measures TTFT, token latency, and throughput while fixing the model, server, client, and workload.

A self hosted llm performance benchmark is credible only when the model, serving arguments, hardware and software, client, dataset, and traffic pattern are fixed before the run. Measure first-token delay, token-by-token latency, and throughput under those same conditions, and retain the raw report and configuration so another operator can repeat the test. Separate server performance from startup and application-layer effects; otherwise a comparison can describe the whole request path without identifying which layer changed.

Prepare a repeatable self hosted llm test matrix

Start with the system under test, or SUT. The MLPerf Inference Rules define it as the defined set of hardware and software resources being measured. Record the model identity and exact files or URI, tokenizer, vLLM build, GPU model and count, host CPU and memory, driver and runtime, container image, server command, client location, and network path. Freeze those values for the baseline, then change only the setting under evaluation.

The vLLM serve CLI reference distinguishes memory, context, and batching controls. Record the effective values rather than assuming that an omitted option still represents your intended configuration:

--gpu-memory-utilization YOUR_VALUE
--kv-cache-memory-bytes YOUR_VALUE
--max-model-len YOUR_VALUE
--max-num-seqs YOUR_VALUE
--max-num-batched-tokens YOUR_VALUE

--gpu-memory-utilization is the GPU-memory fraction used by the model executor. Its documented range is 0 to 1, and its default is 0.92. This is a per-instance limit, even when another vLLM instance uses the same GPU. Setting --kv-cache-memory-bytes establishes the KV cache size per GPU and overrides the GPU-memory-utilization setting when present.

--max-model-len covers prompt and output length. If it is unset, vLLM derives the value from the model configuration; -1 or auto selects the largest model length that fits in GPU memory. --max-num-seqs caps sequences processed in an iteration, while --max-num-batched-tokens caps tokens processed in an iteration.

If startup performance is part of the report, separate warm and cold cache states. The vLLM optimization guidance says torch.compile artifacts are stored under the following setting and default path:

VLLM_CACHE_ROOT
~/.cache/vllm

Those artifacts can be copied between machines or included in a container image. Passing a previously logged --kv-cache-memory value on the next boot skips memory profiling and the CUDA-graph memory-estimation pass. Do not mix cached and uncached startup results under one benchmark label, and do not assume --kv-cache-memory and --kv-cache-memory-bytes are interchangeable without checking the current CLI documentation.

Run and control the benchmark workload

The vLLM Benchmark CLI documentation positions vllm bench for feature evaluation and regression testing, and recommends GuideLLM for production vLLM servers. It also supports ShareGPT for online and offline benchmarks. Download the documented dataset file with:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

For an online test, start the server first and point vllm bench serve at its endpoint:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

For offline throughput measurement, use:

vllm bench throughput \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset-name sonnet

These commands exercise different execution paths. Do not present offline throughput as an online API result: the online command includes the deployed endpoint, while the offline result answers a narrower throughput question.

For controlled traffic and report interpretation, the same vLLM Benchmark CLI reference documents these controls:

--request-rate
--max-concurrency
--burstiness
--ramp-up-strategy

--request-rate sets the target request-generation rate in requests per second. inf requests maximum throughput and is the default, which sends requests immediately. A finite value provides controlled load simulation. --max-concurrency caps outstanding requests; leaving it unset allows unlimited concurrency. --burstiness shapes arrivals with a Gamma distribution: lower values create burstier traffic, higher values create more uniform traffic, and 1.0 represents Poisson traffic. Request rate can also ramp linearly or exponentially during a run.

The report records request throughput in requests per second, output-token throughput, and total-token throughput. Its timing fields have specific meanings:

  • TTFT is measured at the client from sending the request until the first streamed output arrives.
  • ITL records the gap between consecutive streamed outputs and is aggregated over successful requests.
  • TPOT is calculated once per request, excludes the first token, and is then aggregated across requests.

With speculative decoding, a streamed output can contain several tokens. ITL does not insert zero-duration gaps for tokens contained in the same output, so ITL and TPOT should not be expected to converge. Metric terminology is also not standardised across benchmarking tools; compare definitions before comparing values.

Repeating the benchmark against the same server can reuse prompts in its prefix cache and inflate throughput. Treat such a rerun as potentially cache-assisted rather than as an independent confirmation of the earlier result.

Use MLPerf when governed comparability is required

A generic vllm bench run is not automatically an MLPerf result. The MLPerf Inference Rules require every submission to use LoadGen, and the selected scenario changes both traffic delivery and the reported metric:

  • Server: LoadGen sends queries according to a Poisson distribution, applies a benchmark-specific latency constraint, and reports the maximum Poisson throughput supported.
  • Offline: LoadGen sends all samples at the start in a single query, applies no latency constraint, and reports measured throughput.
  • Single stream: LoadGen sends the next query after the previous query completes and reports a 90% early-stopping latency estimate.

For LLM benchmarks, MLPerf collects TTFT and TPOT. Server latency constraints are expressed as TTFT/TPOT limit pairs that differ by category. Record the applicable category and constraint before interpreting a result, rather than treating every MLPerf LLM result as directly interchangeable.

Include FastAPI in the measured path when applicable

If the benchmark endpoint passes through FastAPI, its streaming behaviour belongs in the system definition. The FastAPI streaming documentation explains that declaring the following response class lets a path function yield chunks individually:

response_class=StreamingResponse

With StreamingResponse, the application is responsible for producing and encoding the bytes it sends. Because TTFT ends when the client receives its first streamed output, application chunk production and encoding can affect the path being measured.

Container topology can also change the application layer. The FastAPI container guidance says that when Kubernetes handles replication at cluster level, the container should run a single Uvicorn process without multiple workers. It also recommends the exec form of CMD so FastAPI can shut down gracefully and run lifespan events. The documented sample command is:

CMD [ "fastapi", "run", "app/main.py", "--port", "80" ]

Benchmark the command and process layout that are actually deployed rather than substituting this sample configuration.

Check the run and troubleshoot common failure modes

Before accepting a result, verify that the saved SUT manifest matches the running server and that the client used the intended endpoint, dataset, and traffic settings. Confirm that the report contains the metrics promised by the chosen method, then preserve the command, configuration, raw output, and relevant server and client logs.

If throughput rises unexpectedly, inspect prefix-cache effects, request-rate settings, concurrency, and application processing before attributing the change to hardware or engine tuning. If latency differs between tools, compare TTFT, ITL, and TPOT definitions instead of matching labels by name. If startup time is included, label the compile-cache and memory-profiling state. A valid comparison changes the selected engine argument, loader, hardware, or workload while keeping the remaining SUT fields fixed.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources