AI DeploySelf-hosted LLMGet it deployed

Choose a Modal GPU for LLM Inference

Modal recommends a 48 GB L40S for inference; compare documented GPU memory, configure vLLM controls, and review billed time before deploying an LLM Function.

For ai model deployment on Modal, start with L40S for an initial inference Function. Modal’s GPU acceleration guide recommends L40S for inference and attaches a GPU through the gpu argument of the Function decorator. If the model, context, and batching setup do not fit the selected card, move to a larger documented request, then verify the endpoint defined by your app and its billable runtime before adding concurrency.

Decide before ai model deployment begins

Use a fixed selection order:

  • Identify the model revision, runtime format, and any quantization choice from the model project’s current documentation.
  • Record the longest context and greatest batch your application must handle.
  • Estimate the working memory needed for weights and runtime state.
  • Decide whether one GPU is sufficient or whether tensor parallelism is required.
  • Record a preferred GPU, any ordered fallback, and the billable window you intend to test.

A GPU’s nominal memory is only one constraint. A configuration can fail when model weights, context, and active sequences must fit together, even if the model’s base file appears smaller than the card’s memory capacity.

Compare documented GPU memory

This focused comparison uses memory facts that directly affect the initial decision.

Modal option Documented memory Workload-fit question
L4 24 GB GPU memory, according to NVIDIA’s L4 specification Can your exact model configuration and vLLM limits fit within this capacity?
L40S 48 GB GDDR6 with ECC, according to NVIDIA’s L40S specification Does it hold the required working set while preserving the necessary context and batch size?

This is not a speed ranking or a complete Modal catalogue. Check the current Modal and model documentation before selecting an option not covered here.

Configure the Modal Function

Modal’s GPU acceleration guide supplies the remaining request rules. Append :n to request multiple GPUs; most types support up to eight per container, while A10 supports up to four. Requests above two usually wait longer, and all GPUs in a container are attached to the same physical machine.

An H100 request may be upgraded to an H200 without changing the GPU cost, while H100! opts out of that upgrade. An A100 request starts as the 40 GB version and may be upgraded to 80 GB; A100-80GB requests the larger version explicitly. The documented B200+ request can run on B200 or B300 but is billed as B200. B300 requires CUDA 13.1 or later, and GPU use requires a valid payment method.

These are alternative decorator configurations, not blocks to stack:

@app.function(gpu="L40S")

Use the single-GPU form when the complete working set fits.

@app.function(gpu=["H100", "A100-40GB:2"])

This documented fallback example tries H100 first, then the listed A100 configuration. The :2 count belongs to the second option.

@app.function(gpu="H100:8")

This requests multiple GPUs. Do not treat the documented maximum as a normal starting target; review the longer-wait and co-location behavior before using it.

Fit vLLM to GPU memory

The vLLM memory-conservation guide identifies max_model_len as the context-length control and max_num_seqs as the maximum batch-size control. Reducing either can lower memory usage. The same guide says tensor_parallel_size can split a model across several GPUs, but every tensor-parallel process reads the whole checkpoint, so disk-loading time grows as the parallel size increases.

Use the exact setting names supported by the vLLM version in your application:

max_model_len
max_num_seqs
tensor_parallel_size

Set the context and sequence limits to what the application can support, not merely what the GPU advertises. Lowering the context limit can exclude longer prompts; lowering max_num_seqs reduces batching capacity. Use tensor parallelism only when the single-GPU memory budget is insufficient.

The guide also directs device selection through CUDA_VISIBLE_DEVICES before vLLM starts rather than through torch device calls:

CUDA_VISIBLE_DEVICES=YOUR_VALUE

Bound concurrency after memory

Modal’s Input concurrency guide says each container handles one input at a time by default. It documents @modal.concurrent(max_inputs=100) as the syntax for enabling concurrent inputs, and identifies continuous batching with vLLM as a GPU use case. When demand exceeds the input limit, Modal adds containers, although extra inputs may wait while those containers cold start.

@modal.concurrent(max_inputs=100)

The documented value is an example, not a default or performance promise. Modal suggests deriving target_inputs from the desired latency and max_inputs from resource constraints, including avoiding GPU out-of-memory failures. Synchronous Functions execute concurrent inputs on separate threads, so their implementations must be thread-safe.

Calculate the billable window

Modal’s pricing documentation meters CPU, memory, and GPU compute by the second. Billable application time includes loading, input processing, and an idle keep-alive window that defaults to 60 seconds after the last input. Once the application has no containers, charges stop. CPU and memory are billed at the higher of requested and actual use.

Before deployment, read the current page or quote and record:

  • The GPU rate for each candidate.
  • CPU and memory rates.
  • The included compute shown for your plan.
  • Load, processing, and keep-alive seconds for each container.
  • Any additional charges or credits shown on your account.

Use placeholders rather than inserting an unverified price:

GPU_RATE = YOUR_VALUE
LOAD_SECONDS = YOUR_VALUE
PROCESSING_SECONDS = YOUR_VALUE
KEEP_ALIVE_SECONDS = YOUR_VALUE
CPU_MEMORY_COST = YOUR_VALUE
OTHER_COST = YOUR_VALUE
CREDIT = YOUR_VALUE

BILLABLE_SECONDS = LOAD_SECONDS + PROCESSING_SECONDS + KEEP_ALIVE_SECONDS
GPU_ESTIMATE = GPU_RATE * BILLABLE_SECONDS
TOTAL_ESTIMATE = GPU_ESTIMATE + CPU_MEMORY_COST + OTHER_COST - CREDIT

Apply the calculation across the containers expected to serve the workload. Do not derive a quote from GPU memory or an upgrade rule; use the provider’s current rate and your own bill or quote.

Verify configuration, request, and usage

The official configuration examples do not provide a universal inference URL, so use the endpoint already defined by your application rather than inventing a path.

Check the deployment in this order:

  • Confirm that the Function uses the intended gpu value.
  • For a fallback list, confirm that the preferred option appears first.
  • For multiple GPUs, confirm the :n count and review the documented placement behavior.
  • Confirm that max_model_len, max_num_seqs, and device selection match the model configuration.
  • Send the smallest valid request to your own endpoint and check the expected response and absence of an out-of-memory failure.
  • Increase context, batching, or concurrency one control at a time.
  • Reconcile recorded usage with the expected loading, processing, and keep-alive window.

A successful request confirms that the path works, but it is not a latency or throughput result.

Common failure points

  • Choosing from the GPU name before calculating the full working set.
  • Treating an ordered fallback as a guarantee of the first-choice hardware.
  • Requesting the maximum GPU count without considering waits and same-machine placement.
  • Raising context length, batch size, and input concurrency together.
  • Selecting devices through torch before vLLM starts instead of using CUDA_VISIBLE_DEVICES.
  • Enabling concurrent synchronous inputs before checking thread safety.
  • Reviewing active processing time but overlooking the billable keep-alive window.

No commands were run or measurements made for this article. These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources