AI DeploySelf-hosted LLMGet it deployed

Deploy AWQ, GPTQ, or FP8 Models with vLLM

Prepare AWQ, GPTQ, and FP8 models for vLLM by matching hardware support, choosing documented load options, and checking memory limits.

Deploy AWQ, GPTQ, or FP8 models with vLLM by matching the checkpoint format to the target GPU, using that format’s documented load path, and setting context and memory controls before accepting requests. For this ai model deployment, treat AWQ, GPTQ, and FP8 as separate branches rather than interchangeable labels. Work through hardware compatibility, model loading, memory limits, and output validation in that order.

Check target GPU and checkpoint compatibility

The vLLM quantization overview describes quantization as trading model precision for a smaller memory footprint and points to llm-compressor as a starting point for FP8, INT8, INT4, and other formats.

The hardware matrix in the vLLM quantization overview maps Volta to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0; it lists AWQ as unsupported on Volta and AMD GPUs but supported on Turing, Ampere, Ada, Hopper, Intel GPU, and x86 CPU, and marks llm-compressor FP8 W8A8 for Ada, Hopper, and AMD GPU columns. The documentation warns that this chart can change as support evolves.

Before converting or downloading a checkpoint, record:

  • The exact model identifier and checkpoint format.
  • The target GPU architecture and available GPU memory.
  • Whether you need offline LLM(...) loading or an online vllm serve process.
  • Whether the model is already quantized or still uses BF16 or FP16 precision.

Keep the original checkpoint if you need a quality baseline. Do not infer kernel support from GPU memory capacity alone.

Set context, batch, and memory limits

The vLLM serve CLI reference defines --max-model-len as the context length covering prompt plus output, --max-num-seqs as the maximum number of sequences processed in one iteration, and --max-num-batched-tokens as the maximum number of tokens processed in one iteration; if the context length is omitted, vLLM derives it from the model configuration, while -1 or auto selects the largest length that fits in GPU memory.

The same vLLM serve CLI reference documents --gpu-memory-utilization as the GPU-memory fraction used by the model executor, with a default of 0.92. It is a per-instance limit: the documentation gives two instances on one GPU as an example where each can use 0.5. Setting --kv-cache-memory-bytes specifies KV-cache memory per GPU and overrides gpu_memory_utilization for that sizing decision.

Use these documented flags as fragments in the applicable model command:

--max-model-len YOUR_VALUE
--max-num-seqs YOUR_VALUE
--max-num-batched-tokens YOUR_VALUE
--gpu-memory-utilization YOUR_VALUE

Replace each YOUR_VALUE from your workload rather than copying an arbitrary example. Use an explicit context length when you need a predictable cap. If you set --kv-cache-memory-bytes, do not assume the GPU-memory fraction still controls the KV-cache allocation.

Deploy an AWQ checkpoint

The vLLM AWQ guide states that AutoAWQ is deprecated and that the functionality has moved into llm-compressor. Its documented Python loading example is:

llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="auto_awq")

Treat this as an AWQ loading example, not a model recommendation. Use auto_awq only when the checkpoint is actually AWQ and its hardware entry is supported. If the model source identifies the checkpoint as GPTQ, FP8, or another format, return to the compatibility matrix and use that format’s own path instead of forcing the AWQ setting.

Deploy a GPTQ checkpoint

The vLLM GPTQModel guide says GPTQModel can produce 4-bit and 8-bit models that vLLM can run; compatible GPTQModel checkpoints can use the Marlin and Machete kernels on Ampere and Hopper GPUs.

Treat those kernel names as execution details, not launch arguments. Do not copy auto_awq or fp8_per_tensor into a GPTQ command without the checkpoint’s current documentation.

Prepare the W4A16 branch

The vLLM INT4 W4A16 guide says this computation requires NVIDIA compute capability above 8.0, covering Ampere, Ada Lovelace, Hopper, and Blackwell. It gives this exact modifier recipe:

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head"],
)

The same INT4 W4A16 documentation says weight quantization needs sample data to estimate updates and calibrated scales, preferably resembling the deployment data. Its recommended starting point is 512 samples, increasing that amount if accuracy drops.

The recipe is for the W4A16 branch; it is not a universal 8-bit GPTQ launch command. Because the checkpoint identifier and producer options vary, obtain the final model argument and launch syntax from that checkpoint’s current project documentation rather than inventing them.

Deploy an FP8 checkpoint

The vLLM FP8 W8A8 guide distinguishes hardware-accelerated W8A8 operation on GPUs such as NVIDIA H100 and AMD MI300X from weight-only W8A16 operation on Turing and Ampere GPUs through Marlin kernels. It says NVIDIA FP8 computation requires compute capability >= 8.9 and names Ada Lovelace, Hopper, and Blackwell.

The same FP8 W8A8 guide says to install llm-compressor and keep it in a separate environment from vLLM; its recommended FP8_DYNAMIC recipe targets Linear, ignores lm_head, and needs no calibration data. It loads a saved model by its local directory and can also quantize a BF16 or FP16 model at load time through the command-line or constructor option.

The documented installation example is:

(venv-llm-compressor) pip install llmcompressor

The recommended recipe fragment is:

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

After the documented save process creates an FP8 model directory, the documented loading example is:

llm = LLM("./Meta-Llama-3-8B-Instruct-FP8-Dynamic")

For load-time quantization of a BF16 or FP16 model, use the applicable documented fragment:

--quantization="fp8_per_tensor"

quantization="fp8_per_tensor"

These are exact recipe and loading excerpts, not a complete conversion program. Take the surrounding model-loading and save calls from the current official guide, and do not add AWQ or GPTQ quantization values to the FP8 path.

Validate the ai model deployment

Start with the exact loading line for the selected format. A completed load confirms that the model-loading path ran, but it does not establish that every output is correct.

Before writing a client, consult the current vLLM serving documentation for the exact endpoint, HTTP method, and request schema. Do not guess an OpenAI-compatible route or a health-check path. Send a small set of fixed, representative prompts through that documented interface and compare the results with your intended baseline under the same generation settings.

Check the deployment in this order:

  • Loading: The documented constructor or serving command completes without a checkpoint, format, or kernel error.
  • Serving: A request using the current endpoint and schema returns without a transport or schema failure.
  • Output quality: Representative tasks produce the expected language, structure, and task behavior when compared with the baseline.
  • Configuration: Record the model identifier, quantization format, GPU architecture, context length, sequence limits, token limits, and memory settings.
  • Repeatability: Repeat the same checks after changing one memory or serving parameter at a time.

A successful startup or a single valid response does not establish accuracy across your workload.

Where deployment usually goes wrong

  • Wrong format branch: Do not apply an AWQ, GPTQ, or FP8 setting merely because another model used it successfully. Start from the checkpoint metadata and compatibility matrix.
  • Stale hardware assumption: Recheck the current chart before deployment because supported quantization and hardware combinations can change.
  • Recipe mistaken for a complete script: The GPTQ and FP8 recipe objects shown here are configuration fragments. Use the current guide for the surrounding conversion and save workflow.
  • Environment collision: Recreate the documented separation between the quantization environment and the vLLM environment if model preparation and serving interfere.
  • Memory blind spot: Record the effective context, batch, token, and GPU-memory settings. An automatically selected context or an unchanged default may not match the intended workload.
  • Invented validation path: Obtain the request endpoint and schema from current documentation. Do not treat a successful process launch as proof that the client contract or model output is correct.

These steps are compiled from the official documentation linked in the text; nothing was run or measured for this article. Check the commands against the current project version before use.

Sources