Configure FP8 in vLLM on an H100

Configure FP8 weights and activations in vLLM on H100, then validate your own model’s precision and serving performance with documented checks.

On an H100, configure FP8 either by loading an LLM Compressor artifact in vLLM or by passing --quantization="fp8_per_tensor" when loading a BF16/FP16 model; vLLM's FP8 W8A8 guide documents hardware-accelerated W8A8 on H100 and no calibration data for that load-time flow. For this self hosted llm, choose either a prequantized directory or the shorter load-time setup, then retain one baseline configuration for comparison. Validate precision and serving performance against your own model workload rather than treating a general documentation statement as an expected result.

Check H100 support and the exact card variant

NVIDIA's H100 specifications list 80GB of memory for H100 SXM and 94GB for H100 NVL. The listed maximum thermal design power is up to 700W and configurable for SXM, or 350-400W and configurable for the PCIe NVL card. NVIDIA also lists SXM and PCIe dual-slot air-cooled form factors.

Record the precise card or rented instance configuration before estimating capacity. The vLLM quantization compatibility notes map Hopper to SM 9.0, but warn that the compatibility chart can change as vLLM evolves. Recheck the current chart for the model, quantization method, GPU generation, and vLLM version you intend to use.

Prepare the self hosted llm environment on H100

The official FP8 workflow calls for separate Python environments for vLLM and LLM Compressor because they may not work together. It provides the LLM Compressor installation command, uses FP8_DYNAMIC for Linear layers while ignoring lm_head, and does not require calibration data for this flow.

Before changing code:

Choose the load-time route for a BF16/FP16 source model, or prepare a separate FP8 artifact with LLM Compressor.
Keep the conversion and serving environments isolated.
Record the vLLM, LLM Compressor, CUDA, driver, and model versions used for the run.
Preserve the original checkpoint and the exact FP8 output directory so a failed validation can be reproduced.

Create an offline FP8 artifact

Install LLM Compressor in its dedicated environment using the documented command:

pip install llmcompressor

The documented modifier configuration is:

recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"], )

This is the modifier configuration rather than a complete checkpoint-conversion script. Follow the current guide's complete model-specific loading and save example to produce the FP8 directory; do not invent a universal checkpoint command for a model that has not yet been checked.

After that process creates a local directory, the documented vLLM loading expression is:

llm = LLM("./Meta-Llama-3-8B-Instruct-FP8-Dynamic")

Replace the example directory with the path your conversion process actually produced. Keep the offline artifact separate from the original BF16/FP16 model so both configurations remain available for an accuracy and serving comparison.

Use load-time FP8 quantization

The vllm serve CLI reference explains that engine arguments are passed to vllm serve. For a BF16/FP16 model, combine that interface with the documented FP8 load-time option:

vllm serve YOUR_MODEL --quantization="fp8_per_tensor"

Replace YOUR_MODEL with the model identifier available in your environment. This route performs the FP8 conversion during loading and therefore does not use the LLM Compressor recipe described above. Do not combine the two workflows merely because both use the term FP8: choose one artifact and one loading path, then document that choice.

Size the vLLM serving instance

The vLLM serving options define --gpu-memory-utilization as the fraction assigned to the model executor. Its documented range is 0 to 1, its default is 0.92, and the limit applies to each vLLM instance. For two instances sharing one GPU, the documentation gives 0.5 as the setting for each instance.

Several related controls affect whether the model and workload fit:

--kv-cache-memory-bytes sets KV-cache memory directly per GPU. When supplied, it takes precedence over the GPU-memory-utilization setting.
--max-model-len covers both prompt and output tokens. If omitted, vLLM derives the value from the model configuration.
-1 or auto for --max-model-len selects the largest model length that fits in GPU memory.
--max-num-seqs limits sequences processed in one iteration.
--max-num-batched-tokens limits tokens processed in one iteration.
--enforce-eager disables CUDA graphs and torch.compile, so keep it available as a diagnostic control rather than the starting configuration.

Start with an explicit workload length where practical. Automatic selection is useful for discovery, but it can make later comparisons harder to explain if available GPU memory or the serving configuration changes.

Verify precision before throughput

Create a BF16 or FP16 baseline from the same source model and retain the same tokenizer, prompts, task set, and generation settings for the FP8 comparison. A successful model load confirms that initialization completed; it does not establish task quality.

For an lm_eval evaluation, include the documented setting:

add_bos_token=True

The FP8 guide specifically warns that quantized models can be sensitive to the presence of the BOS token, while lm_eval does not add it by default. Take the remaining runner syntax from the current lm_eval documentation rather than assuming that every model or task uses the same command.

Record task-level outcomes, not just an overall impression. Review the workloads your application actually performs, investigate regressions rather than averaging them away, and keep failed or truncated responses visible in the record. Only after the precision comparison is understood should the same configurations proceed to performance testing.

Benchmark the running vLLM server

The Benchmark CLI documentation identifies ShareGPT as a supported dataset for online and offline benchmarks and provides this download command:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

With the server running, the documented online benchmark form is:

vllm bench serve \
    --backend vllm \
    --model NousResearch/Hermes-3-Llama-3.1-8B \
    --endpoint /v1/completions \
    --dataset-name sharegpt

That example uses the /v1/completions path. It does not specify a nondefault host or port, so consult the current CLI reference for your connection rather than copying an invented option.

The report includes request throughput and output-token and total-token throughput. It defines time to first token, or TTFT, as the client-side interval from sending a request to receiving its first streamed output. Inter-token latency, or ITL, records the gaps between consecutive streamed outputs across successful requests. Time per output token, or TPOT, is calculated once per request without the first token and then aggregated.

For a useful comparison, keep the model, dataset, endpoint route, input and output lengths, concurrency, and request mix aligned. Run the BF16/FP16 and FP8 configurations under the same H100 environment, preserve each report, and compare the distributions rather than selecting a single favorable run.

Where the setup usually goes wrong

Start troubleshooting at the stage that changed:

Environment collision: Keep vLLM and LLM Compressor in their separate environments, as required by the FP8 workflow.
Wrong FP8 path: The load-time option is documented for BF16/FP16 input; an offline FP8 model is loaded from the directory produced by LLM Compressor.
Memory pressure mistaken for FP8 incompatibility: vLLM's memory-conservation notes explain that CUDA graphs take extra GPU memory. Account for that overhead before changing the quantization method.
Uncontrolled context selection: Record whether --max-model-len came from the model configuration, an explicit workload limit, or auto.
Stale compatibility assumptions: Recheck the current quantization chart instead of relying on an older H100 support matrix.
Confounded benchmarks: If precision, context length, concurrency, or request composition changed at the same time as FP8, the report cannot isolate the quantization effect.

These steps are compiled from the official documentation linked in the text, not from a hands-on run, and should be checked against the current versions before use.