Compare vLLM, Replicate, and Modal Deployments

Compare how vLLM, Replicate, and Modal divide deployment control across serving memory, billing scope, GPU selection, and operational checks.

Compare vLLM, Replicate, and Modal at the control boundary: vLLM's vllm serve options configure an online server; Replicate's billing rules separate public models, private models, and deployments; and Modal's GPU options select hardware for a Function. For ai deployment, the practical choice depends on whether you need serving-memory controls, provider-managed queues and billing, or container-level GPU allocation—not on a claim that one option fits every workload. The cited documentation does not define a common launch path, so the commands and fragments below stay within documented behavior; resolve installation, model compatibility, and endpoints in each project's current documentation.

Questions to settle for ai deployment

Before changing code, write one deployment record:

vLLM: model identifier, GPU count and interconnect, prompt-plus-output context, batching policy, and ownership of the server process.
Replicate: model or deployment class, expected active and idle periods, queue tolerance, hardware, and billing method.
Modal: required GPU, acceptable fallback order, GPUs per container, and CUDA compatibility.

Treat every unresolved item as a verification task. A generic example does not establish model compatibility, sufficient memory, acceptable queueing, or billing treatment.

vLLM: set memory and parallelism

The vllm serve CLI reference defines --max-model-len as the context covering both prompt and output; if omitted, vLLM derives it from the model configuration. It defines --max-num-seqs as the sequence limit for one iteration and --max-num-batched-tokens as the token limit. The documented default for the per-instance --gpu-memory-utilization is 0.92.

Use this as a fill-in template rather than a model-specific configuration:

vllm serve YOUR_VALUE \
  --tensor-parallel-size YOUR_VALUE \
  --pipeline-parallel-size YOUR_VALUE \
  --max-model-len YOUR_VALUE \
  --max-num-seqs YOUR_VALUE \
  --max-num-batched-tokens YOUR_VALUE \
  --gpu-memory-utilization YOUR_VALUE

The first placeholder is the model identifier; each later placeholder takes the corresponding value from your model, workload, and hardware plan. Replace every placeholder before use.

The vLLM optimization and tuning guide describes tensor parallelism as sharding model parameters across GPUs within each model layer and as the common single-node strategy. Pipeline parallelism instead distributes different layers across GPUs, which process their parts in sequence. A deployment using N GPUs needs at least 2 + N physical CPU cores under the documented minimum.

For multiple nodes, the project's parallelism and scaling guide gives this rule: set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes. Choose these values together with context and batch limits; increasing GPU parallelism does not remove the need to check model fit.

If startup matters more than steady-state decoding, --enforce-eager skips compilation and CUDA-graph capture. vLLM documents this as the fastest-possible startup path, but with lower steady-state decode performance.

Replicate: classify billing before deployment

Start with the billing class because “model” does not represent one charge case. The Replicate billing guide says public models bill only active processing time, while setup and idle time are free; public models also use a shared hardware pool and queue by default. Most private models use dedicated hardware without a shared queue, while fast-booting fine-tunes are the documented exception and bill only active time. Private models otherwise bill setup, idle, and active time; deployments do the same.

A deployment's minimum instance count defaults to 0 and can be customized. After a gap in requests, an instance remains idle for a few minutes instead of shutting down immediately. Failed runs are not charged, but failed or cancelled private-model and deployment runs still bill their active instance time; a cancelled official-model run may also be charged. If a model calls other models, include both its compute time and the downstream models' compute.

Before estimating, read Run time and cost on the model page and identify the applicable charge type. The official-model documentation says official models use output metrics such as images, video seconds, and input or output tokens instead of hardware and run time.

Use only the line items and rates shown on the relevant model page, quote, or bill:

PUBLIC_ACTIVE_COST = ACTIVE_TIME * ACTIVE_TIME_RATE
DEPLOYMENT_COST = (SETUP_TIME + IDLE_TIME + ACTIVE_TIME) * DEPLOYMENT_RATE
OFFICIAL_MODEL_COST = SUM(APPLICABLE_METRIC_COUNT * PUBLISHED_METRIC_RATE)

Apply only the formula that matches the model. Expand the official-model formula only for metrics actually listed for that model, and add downstream-model compute when the model invokes other models.

Modal: select GPU allocation

Modal's GPU acceleration guide attaches a GPU to a Function through the gpu decorator argument and requires a valid payment method. Its documented values are T4, L4, A10, L40S, A100, A100-40GB, A100-80GB, RTX-PRO-6000, H100/H100!, H200, B200/B200+, and B300.

The allocation rules matter when choosing a value:

Append :n to request multiple GPUs in one container. Most documented GPU types support up to 8; A10 supports up to 4. Requesting more than 2 usually creates longer waits, and all GPUs in a container are attached to the same physical machine.
For inference, Modal's guide recommends starting with L40S and identifies its GPU memory as 48 GB.
The gpu argument accepts an ordered fallback list. Modal attempts the preferred type before moving to less-preferred options.
An H100 request may be upgraded to H200 at no extra cost; H100! prevents that upgrade. An A100 request may be upgraded to an 80 GB A100 at the same cost, while A100-80GB requests that version explicitly. B200+ permits B200 or B300 allocation and is billed as B200.
B300 requires CUDA 13.1 or later, so verify the container image and libraries first.

These are argument fragments, not a complete Function:

gpu="L40S"
gpu="H100:8"
gpu=["H100", "L40S"]

Use one selection form in the actual decorator. The cited GPU guide does not provide application scaffolding, a universal health endpoint, a cold-start duration, or a general autoscaling policy; obtain those details from the current Modal Function documentation.

Check what the documentation can verify

For vLLM, the parallelism and scaling guide explains that the startup line GPU KV cache size reports the total number of tokens that fit in the GPU KV cache at once. Use that figure as a capacity check against your context and concurrency settings, not as a performance measurement.

For Replicate, the cited billing and official-model pages do not define a universal health endpoint. Confirm the intended model or deployment class and the hardware shown on the model page, then obtain the request, status, and health-check procedure from the current deployment documentation.

For Modal, record the requested GPU, fallback order, container count, and CUDA requirement. The GPU guide does not settle runtime readiness, so use the current Function documentation for the health procedure and scaling behavior. A configuration record shows intent; it does not replace a documented runtime response.

Common failure points

Follow vLLM's preemption guidance when KV-cache pressure becomes frequent. Preemption occurs when KV-cache space runs out, and affected requests are recomputed later. The documented remedies are to increase gpu_memory_utilization first or decrease max_num_seqs or max_num_batched_tokens. If chunked prefill is disabled, max_num_batched_tokens must also be greater than max_model_len.

For Replicate, do not reuse the public-model active-time formula for a private model or deployment without checking the exceptions and setup, idle, active, and downstream lines. For Modal, review exact GPU behavior, fallback order, same-machine placement, wait guidance, and CUDA compatibility rather than selecting only by GPU name.

These steps are compiled from the official documentation linked in the text; no deployment was run or measured for this article, and the current version should be checked before use.