AI DeploySelf-hosted LLMGet it deployed

Mounting Modal Volumes for Faster Model Loading

Learn how to configure a Modal Volume, reuse model weights, and avoid repeated downloads in an open-source LLM deployment workflow.

For faster model loading in ai model deployment, store model weights in a Modal Volume, mount it at a stable path such as /data, and load the weights when a container starts. Modal’s model-weight guidance recommends Volumes, while its cold-start guidance says downloading weights ahead of time avoids another download at boot; neither choice guarantees a fixed cold-start improvement.

Prepare storage for ai model deployment

Before editing, record four values:

  • The local directory containing the model weights.
  • The name of the Modal Volume.
  • The path where the Volume will be mounted inside the container.
  • The model identifier used by your existing loading code.

Treat these as one contract. The upload destination, mount path, and path passed to the model loader must resolve to the same files. Keep the weight directory structure unchanged unless your loader is configured to expect a different layout.

Create and populate the Volume

Modal’s Volumes guide documents CLI and code-based creation. The basic CLI operation is:

modal volume create

To create a v2 Volume, add the documented version flag:

modal volume create --version=2

Alternatively, code can declare a Volume lazily:

vol = modal.Volume.from_name("my-volume", create_if_missing=True)

With create_if_missing=True, the Volume is created if the named Volume does not already exist. Use either creation route, but keep the resulting name consistent with later upload and mounting code.

Choose the version carefully. The current documentation describes Volumes v2 as Beta, says Modal does not recommend it for mission-critical data, and states that there is no automated v1-to-v2 upgrade; data must be copied to a new v2 Volume.

Once the weights are available locally, use the documented CLI upload pattern:

modal volume put model-weights-vol path/to/model path/on/volume

The model-weight upload example takes a local source path and a destination on the Volume. Replace both path arguments with the locations used by your project, and make the resulting directory structure match what the loader expects.

Mount the Volume and control file visibility

Modal’s Volumes documentation defines attachment as a mapping between a container path and a Volume in the function decorator’s volumes argument:

@app.function(volumes={"/data": vol})

Here, /data becomes the stable root used by the loading code. If the serving process only needs to read the weights, the documented read-only form is:

volumes={"/data": volume.with_mount_options(read_only=True)},

Several common failures follow from the Volume’s visibility and concurrency rules:

  • Changes are not visible. A container receives the Volume’s latest state when it is created. It does not see later commits until it calls .reload(). Changes made inside a container must be committed before they become visible outside it. Background commits run every few seconds, with a final commit on container shutdown.
  • Reloading reports volume busy. Calling .reload() while files on the Volume remain open causes that documented error. Close those files before asking the container to refresh its view.
  • Concurrent writers overwrite data. Concurrent modifications to the same file follow last-write-wins behavior, and distributed file locking is not supported. Avoid treating the Volume as a synchronized database for overlapping writers.
  • Loading remains slower than expected. Volumes are designed for up to 2.5 GB/s of bandwidth, but Modal explicitly says actual throughput is not guaranteed and may be lower depending on network conditions.

Load weights during container startup

Modal lifecycle hooks require class syntax through @app.cls. Put one-time model initialization inside a method decorated with @modal.enter. The container lifecycle documentation says this handler runs when a new container starts and is intended for initialization such as loading model weights.

Move the existing weight-loading operation into that startup method instead of running it inside every request handler. Read from the same /data path supplied to volumes, and remove any remaining startup-time download when the required files are already present.

This arrangement reuses the stored weights across container starts, but it does not turn the loading process into a persistent in-memory cache. Each new container still has to complete its startup initialization before it can handle normal work.

Check the deployment and benchmark the endpoint

Before running a serving benchmark, verify three alignments:

  • The CLI upload destination contains the expected weight files.
  • The function maps those files beneath the path read by the loader.
  • The loader runs during @modal.enter, rather than being repeated for each request.

With a vLLM server already running, the vLLM benchmark documentation provides an online benchmark example for its documented model and endpoint:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --request-rate inf

ShareGPT is supported for online and offline benchmarks and is obtained from Hugging Face. The --request-rate option controls requests per second; inf requests maximum throughput and is also the documented default. Use a finite value when you want controlled request generation instead.

The resulting report provides serving metrics rather than a direct Volume-loading timer. It records request throughput and output-token and total-token throughput. Time to first token, or TTFT, runs from sending a request to receiving its first streamed output. Inter-token latency measures gaps between consecutive streamed outputs, while time per output token is calculated once per request without the first token.

Avoid treating repeated runs against the same server as independent loading evidence. The vLLM documentation warns that repeated vllm bench serve runs can reuse prompts left in the prefix cache and inflate throughput. Use the report to verify serving behavior and establish a serving baseline, not to claim an unmeasured startup-time improvement.

Separate weight loading from cold starts

A mounted Volume addresses repeated model downloads, but it does not remove every source of cold-start latency. Modal defines a cold start as the need to start a new container when no ready container is available. Delays can come from waiting for a container to become warm and from initialization performed for the first invocation.

Modal states that containers boot in about one second, but global-scope code and @modal.enter methods must finish before the container is considered warm. Moving model loading into @modal.enter therefore relocates that work into the warm-up period; it does not make the initialization free of latency.

Warm-container capacity is controlled separately through scaledown_window, min_containers, and buffer_containers. A longer scaledown_window reduces the chance of a later cold start but keeps resources billable while containers remain idle. min_containers establishes a floor so a Function does not scale to zero, while buffer_containers adds capacity while the Function is active. These controls manage readiness and queueing; the Volume manages where model weights are stored.

Documentation and verification note

No commands or workloads were run or measured for this article. These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources