Measure Cross-Region Latency for LLM Inference

Compare vLLM inference across regions with controlled client-side tests of TTFT, TPOT, inter-token latency, throughput, routing, and load settings.

For a self hosted llm, measure cross-region latency at the client that sends inference requests, not from the distance between Region labels. Run the same streaming workload against every candidate endpoint with the model, vLLM configuration, dataset, request rate, and concurrency held constant, then compare TTFT, inter-token latency, TPOT, and throughput. Repeat the comparison over time because Amazon Route 53’s latency-based routing documentation says internet latency can change as network connections and routing change, and its AWS latency data can differ significantly for resources outside AWS.

Prepare a repeatable self hosted llm comparison

Create one row for every benchmark run and record:

The measurement client’s network location and route to each endpoint.
The endpoint address and API path.
The model identifier, weight format, quantization, vLLM version, and server configuration.
GPU type and count, if the deployment uses GPUs.
The dataset file, prompt shape, and requested output length.
The load controls: request rate, concurrency, burstiness, and ramp-up strategy.
Run start and end timestamps, errors, and the exact command used.

Keep the client origin and workload fixed while changing only the regional endpoint. If a region requires different hardware or server settings, record that difference and do not present the run as a network-only comparison.

Verify every endpoint with the same workload

Have a running vLLM server at each candidate endpoint before starting the benchmark. The vLLM Benchmark CLI documentation supports ShareGPT for online and offline benchmarks. It also documents custom datasets as JSONL files with a prompt field, selected with --dataset-name custom.

Download the documented ShareGPT file with:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

The documented online invocation is:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

This example fixes the backend, model, completion path, and dataset. Use the same values at every endpoint. The cited example specifies /v1/completions but does not settle remote-host addressing, so use the addressing syntax supported by your installed vLLM version rather than copying an unverified flag.

If ShareGPT does not represent your application, create a JSONL file with one prompt field per entry and select it with --dataset-name custom. Keep that file identical across regions; changing prompts between locations would mix workload effects with network effects.

Hold the load constant and read the report

Arrival rate and concurrency can change queueing behavior even when the model and host are identical. The vLLM benchmark controls and metric definitions provide these documented options:

--request-rate sets the target request generation rate in requests per second. Use inf for maximum-throughput testing; it is also the default and sends all requests immediately. Use a finite value for controlled load.
--max-concurrency limits outstanding requests. If it is omitted, concurrency is unlimited.
--burstiness controls traffic variability with a Gamma distribution. Its documented range is greater than zero; lower values create burstier traffic and higher values create more uniform traffic.
--ramp-up-strategy accepts linear or exponential.

Add only the controls required by your declared test profile:

--request-rate YOUR_VALUE
--max-concurrency YOUR_VALUE
--burstiness YOUR_VALUE
--ramp-up-strategy linear

Replace each YOUR_VALUE from your own test design. Do not compare a maximum-throughput run with a run using a finite request rate as though both represent the same offered load.

Read the report using the same documentation’s definitions:

Report item	What it represents
TTFT	Time from sending a request to receiving its first streamed output.
ITL	Time between consecutive streamed outputs, aggregated over successful requests.
TPOT	Calculated once per request, excluding the first token, and then aggregated across requests.
Throughput	Request throughput together with output-token and total-token throughput.

Speculative decoding requires special care: one streamed output can contain several tokens. ITL records only the gaps between streamed outputs and does not insert zero-duration gaps for tokens inside the same output, so ITL and TPOT can diverge.

Two more cautions come from the same documentation. Repeating vllm bench serve against the same server can reuse prompts left in the prefix cache and inflate throughput. Metric terminology is also not standardized across benchmarking tools, so retain the tool version, exact metric names, command, and report together.

Confirm the comparison is valid

Before accepting a result, check that:

Every run started from the same client location.
Each run reached the intended regional endpoint.
Model, server configuration, dataset, and load settings remained aligned.
The report contains every metric you plan to compare.
Failed or interrupted runs are excluded rather than silently mixed with successful runs.
Repeated runs remain separate so that time-of-day routing changes remain visible.

If any answer is no, label the comparison incomplete rather than inferring that one Region is faster.

Match the benchmark method to the question

For a standardized benchmark definition, review the MLPerf Inference Rules. Every submission must use LoadGen. The rules define latency from the point where LoadGen is scheduled to pass a query to the system under test until a reply arrives. They also define the system under test as a defined set of hardware and software resources subject to performance evaluation.

For LLM benchmarks, MLPerf collects TTFT and TPOT. Its scenarios answer different questions:

Server follows a Poisson query distribution, applies a benchmark-specific latency constraint, and reports the maximum Poisson throughput supported.
Offline submits all samples at the start as one query, has no latency constraint, and uses throughput as the result.
Single stream submits the next query after the previous one completes and uses a 90th-percentile early-stopping latency estimate.

Choose the scenario before running the tool. For your own cross-region report, state explicitly whether the client and inter-region network path are included in the system you are evaluating.

Separate network effects from placement and capacity

AWS’s Regions and Zones guide says Regions are isolated and resources are not replicated across them automatically. It also advises selecting a Region close to specific customers or one that meets legal or other requirements. Create and record each regional deployment explicitly; do not assume that deploying in one Region automatically creates the same resources elsewhere.

Resource availability is a separate precondition. Google Cloud’s GPU locations documentation states that specific GPU model availability varies by region and zone. Confirm that the intended model is available in every candidate location before treating missing capacity as a latency result.

AWS also distinguishes capacity failures from quota failures in its EC2 launch troubleshooting guide. InsufficientInstanceCapacity means there is not enough available On-Demand capacity to fulfill the request, and it can occur when launching a new instance or restarting a stopped one. AWS first suggests waiting a few minutes and retrying; it also documents retrying without a specified Availability Zone and submitting a smaller instance request. InstanceLimitExceeded is different: it indicates that the regional instance limit has been reached. Neither launch failure should be mixed into the client’s inference-latency figures.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.