Hot-Load LoRA Adapters in Multi-Tenant Modal LLMs

Configure trusted dynamic LoRA loading, tenant-to-adapter routing, and replica-aware startup checks for a multi-tenant vLLM service on Modal.

For ai model deployment on Modal, run vLLM with a base model that implements SupportsLoRA, enable LoRA on the server, and use the runtime controls in the vLLM LoRA guide to load, unload, and replace adapters; the guide says production runtime loading should be used only in an isolated, fully trusted environment. Keep tenant identity in an authenticated gateway, map each tenant to an allowlisted adapter name, and reconcile that name across the Modal Function’s replicas because each Function has its own autoscaling container pool. Tenants should select from that mapping rather than submit arbitrary model paths or call the loader directly.

A safe ai model deployment starts with the trust boundary

Separate tenant authorization from model execution. Before starting the vLLM process, create a control-plane registry containing:

YOUR_TENANT_ID mapped to one authorized YOUR_ADAPTER_NAME
The adapter path, compatible base model, and deployment revision
The adapters that must be loaded in each serving replica
The adapter that should be used as the rollback target

Your gateway receives the authenticated tenant, resolves that identity against the registry, and sends only the resulting adapter name to vLLM. Do not treat the incoming request’s model value as authorization to load an arbitrary artifact. Path selection, tenant mapping, and adapter activation are separate decisions.

Keep the runtime loading endpoint behind the gateway and restrict it to trusted application code. This is especially important when several customers share the service: tenant isolation does not turn untrusted adapter paths or control requests into trusted input. The registry should also make the base-model relationship explicit so an adapter cannot be associated with an incompatible model configuration.

Keep requests to one selected adapter. If your application needs different behavior for the same tenant, route it to an adapter that represents that behavior rather than attempting to combine several adapters in one prompt.

Size LoRA capacity before tenants arrive

Choose capacity from the adapters you actually intend to serve. Your largest active adapter rank determines YOUR_MAX_LORA_RANK, while concurrency and batching determine YOUR_MAX_LORAS. In the vLLM LoRA configuration source, max_lora_rank has a default of 16, max_loras has a default of 1, and max_cpu_loras must be at least max_loras. The source also says that a LoRA dtype of auto follows the base model’s dtype.

Start the server with explicit values rather than relying silently on defaults:

vllm serve YOUR_BASE_MODEL \
  --enable-lora \
  --max-loras YOUR_MAX_LORAS \
  --max-lora-rank YOUR_MAX_LORA_RANK \
  --max-cpu-loras YOUR_MAX_CPU_LORAS

Treat max_loras as batch capacity, not as a tenant count. Multiple tenants may use the service, but their requests still consume the configured LoRA and base-model execution capacity. Revisit the rank and CPU-adapter settings whenever the adapter catalog changes; a new tenant registration should not silently redefine the capacity assumptions of an already running server.

Hot-load adapters through protected vLLM endpoints

The documented runtime procedure ties the tenant route to the adapter controls: a request selects an adapter through its model parameter, only one LoRA is applied per prompt, runtime updating is enabled by an environment variable, and adapters have explicit load and unload endpoints.

Enable runtime updates in the environment of the vLLM process before starting it:

export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True

Use this operational sequence after the server is ready:

Send a POST request to /v1/load_lora_adapter with the required adapter name and path.
Confirm that the response is HTTP 200 and contains the documented plain-text success message.
Inspect the /v1/models response. For a LoRA entry, parent identifies the base model and root identifies the adapter artifact location.
Send application requests with model set to the registered adapter name, not merely the base-model name.
To replace weights under an adapter name that is already loaded, use the documented load_inplace=True option.
To release an adapter, send a POST request to /v1/unload_lora_adapter with the adapter’s name or ID.

Perform these operations through an internal loader that first verifies the tenant-to-adapter mapping. A successful load should update your deployment state only after the server reports success. Record the adapter revision together with the load result so that later routing decisions do not depend on an ambiguous mix of old and new adapter files.

The POST that changes one vLLM server is not a tenant-wide configuration broadcast. Your reconciliation process must ensure that every replica allowed to serve a tenant has the required adapter state. Otherwise, requests can succeed on one container and fail on another even though the shared adapter registry looks correct.

Attach the model service to a Modal GPU

A GPU is attached to a Modal Function through the function decorator’s gpu argument. Modal’s GPU acceleration guide says a valid payment method must be on file and documents L40S as its starting recommendation for inference. Select the documented GPU value that fits the base model and adapter configuration rather than copying a hardware choice without checking current requirements.

A cold start occurs when Modal has no ready container and must start a new one. Modal’s cold-start documentation explains that queueing for a warm container and first-invocation initialization can both add latency, and that a container is not warm until its initialization work has completed.

Treat adapter readiness as replica-local state. A new container may have the adapter artifacts on shared storage without having loaded them into its vLLM process. For adapters that must be available immediately, either register stable adapters at server startup or make adapter reconciliation part of the service’s readiness process. Exercise the same rule whenever Modal starts a replacement container.

Also avoid moving required model initialization into the path that handles the first tenant request. Preparing the image and loading stable startup state before routing traffic gives the autoscaler a defined warm-up condition rather than making the first tenant request responsible for discovering missing adapter state.

Stage adapter files through a Modal Volume

Modal’s Volumes guide describes Volumes as distributed storage optimized for write-once, read-many workloads such as model weights. Create the Volume and map it into the GPU Function:

modal volume create YOUR_VOLUME_NAME --version=2

vol = modal.Volume.from_name("YOUR_VOLUME_NAME", create_if_missing=True)

@app.function(gpu="L40S", volumes={"/data": vol})

The --version=2 flag creates a v2 Volume. The volumes mapping attaches that Volume at /data; place adapter artifacts beneath the mounted directory and pass their actual paths to the loading endpoint.

Volume visibility has explicit rules. A container mounts the latest state when it is created, but later commits are not automatically exposed to an existing container. Changes made inside a container must be committed before they become visible outside it, and another container must call .reload() to observe later commits.

Use this order for an adapter update:

Write the new artifact beneath the mounted Volume path.
Commit the change so it is visible outside the writing container.
Reload the Volume in existing serving containers that must read the new artifact.
Call the vLLM loading endpoint with the adapter name and path.
Use load_inplace=True when replacing an adapter that is already loaded under the same name.
Run the tenant acceptance checks before marking the revision as active.

New containers can mount the latest committed state at creation, but existing containers still need to reload after a commit. This distinction prevents a rollout from appearing complete merely because the shared Volume contains the new files.

Verify routing, replacement, and replica readiness

Use an acceptance checklist rather than treating a successful loader POST as the only deployment signal:

Confirm that the base model implements SupportsLoRA.
Confirm that VLLM_ALLOW_RUNTIME_LORA_UPDATING=True was present when the vLLM process started.
Load the intended adapter through the internal control path and require HTTP 200.
Check /v1/models for the expected adapter, base-model relationship, and artifact location.
Send a controlled prompt for each authorized tenant mapping and verify that the selected adapter name appears in the corresponding request.
Test an in-place replacement and repeat the tenant acceptance check.
Verify every serving replica, including a newly created container, before declaring the adapter revision active.
Unload an adapter through the documented endpoint before removing it from your registry, so control-plane and model-server state do not diverge.

Common failures usually fall into recognizable groups. If runtime loading is rejected, check the environment variable, LoRA-enabled startup, model compatibility, and configured rank. If only some requests fail, compare tenant mappings with replica readiness. If a new adapter file is not visible, check the Volume commit and reload sequence. The Modal Volumes documentation also warns that concurrent writes to the same file are last-write-wins and that .reload() can fail with volume busy while files remain open, so serialize publication and close adapter files before reloading.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.