Tune Modal Volumes for LLM Data Workflows

Configure Modal Volumes for LLM assets, verify mounts and reload behavior, and balance shared model weights with safer write patterns.

For ai deployment, tune a Modal Volume by matching its version, mount mode, commit pattern, and model-weight loading to how an LLM workflow reads and writes data. Start with immutable shared assets such as model weights, give every mutable file a clear writer, and verify that readers reload committed state before changing GPU or warm-container settings. This separates storage correctness from compute tuning and gives you a workflow you can inspect instead of tune by guesswork.

Put ai deployment decisions at the data boundary

The version, attachment, consistency, and billing choices below follow Modal’s Volumes guide. Make those choices before changing application code.

Inventory and choose the version

Before configuring a Volume, inventory the files and directories you expect to store, the largest individual file, which jobs only read, which jobs write, and whether two jobs can modify the same file. Also mark the data as mission-critical or disposable. Those answers determine the version, mount mode, and commit design.

Volumes v1 work best with fewer than 50,000 files and directories. They have a hard limit of 500,000 inodes, including files, directories, and symbolic links; file creation beyond that limit fails with ENOSPC. Volumes v2 has no total file-count limit, but each file must be under 1 TiB and a single directory can contain at most 262,144 files.

The guide describes v2 as Beta and says Modal does not recommend it for mission-critical data because it cannot yet guarantee that no data will be lost. There is no automated v1-to-v2 upgrade, so migration requires copying the data to a new v2 Volume.

Create and attach

Create a named Volume from the CLI. Add --version=2 only after deciding that the documented v2 status and limits fit your data.

modal volume create YOUR_VALUE
modal volume create YOUR_VALUE --version=2

For a code-defined workflow, the documented lazy-creation and attachment forms are:

vol = modal.Volume.from_name("YOUR_VALUE", create_if_missing=True)

@app.function(volumes={"/data": vol})

Here, /data is the path mounted inside the container, while create_if_missing=True creates the named Volume if it does not already exist. A consumer that should not modify the shared asset can use the documented read-only mount option:

volumes={"/data": volume.with_mount_options(read_only=True)},

Commit and reload

A successful local write is not enough for another container to see the change. Put write operations in a commit context, and have readers explicitly reload before opening data committed after their mount:

with volume.commit():
    # Write files under /data.
    ...

with volume.reload():
    # Read files under /data after another writer has committed.
    ...

Modal describes background commits every few seconds while a Function or Sandbox executes, followed by a final snapshot and commit when the container shuts down. A container receives the latest state when it is created, but later commits remain invisible until it calls reload().

Catch predictable Volume failures

Check these failure patterns before moving a workflow into production:

Open files can make reload() fail with volume busy; close the relevant handles first.
Concurrent writes to the same file are last-write-wins, and distributed file locking is not supported. Assign one writer or stage changes under separate paths.
For small changes on v1, avoid more than five concurrent commits.
Volumes are not block devices and have no fixed capacity. Tools based on statfs, including df and shutil.disk_usage, return placeholder values rather than a usable capacity check.
Storage usage is calculated from a once-daily snapshot of total storage. Deleted data may still be billed for up to four more days.

Store model weights for one-time initialization

Modal’s model-weight storage guide recommends a Volume as the place to store model weights so multiple Functions can access the shared files. Upload a local weight directory with:

modal volume put model-weights-vol path/to/model path/on/volume

The guide says baking weights into an Image has similar performance, but Volumes provide more flexibility. Mount read-only when the serving process only needs to load the weights.

Use @modal.enter to load the weights once when each new container starts instead of loading them on every call. Keep in mind that methods decorated with @modal.enter cannot receive dynamic arguments, so a workflow that changes models at runtime must account for that restriction.

Tune GPU demand and warm containers

Modal’s GPU acceleration guide attaches GPUs through the gpu function-decorator argument and requires a valid payment method on file. Multiple GPUs use the :n form, such as gpu="H100:8". Most documented GPU types support up to eight GPUs per container, while A10 supports up to four. Requests for more than two GPUs usually have longer waits, and all GPUs assigned to one container are attached to the same physical machine.

Modal also accepts an ordered fallback list:

@app.function(gpu=["H100", "A100-40GB:2"])

The most preferred available type is tried before later entries. If a benchmark must remain on H100 rather than the documented automatic H200 upgrade, request gpu="H100!".

Modal’s cold-start guide defines a cold start as the need to start a container when no ready container exists. Its latency comes from waiting for a container and from first-invocation initialization.

Three settings affect warm capacity:

scaledown_window controls the idle period before shutdown. The documented default maximum idle time is 60 seconds, and the setting can range from two seconds to 20 minutes. Increasing it reduces the chance of a later cold start, but resources used by the idle container are billed.
min_containers establishes a floor so the Function does not scale to zero.
buffer_containers keeps extra capacity while the Function is active, which can help bursty traffic.

An @modal.enter method completes before a container receives inputs, but moving initialization there does not remove its latency; it moves that work into the warm-up period. Where feasible, Modal recommends reading multiple large files concurrently during initialization rather than sequentially.

Schedule refreshes without resetting the cadence

Modal’s scheduling guide says a schedule attached through the schedule parameter becomes active only after the App is deployed. Use modal.Cron for a fixed schedule unaffected by redeployment; modal.Period resets whenever the Function is redeployed.

@app.function(schedule=modal.Cron("0 6 * * *", timezone="America/New_York"))

Deploy the App with the documented command:

modal deploy --name daily_heavy heavy.py

A scheduled writer must still follow the commit and reload rules above. Schedules cannot currently be paused: remove the schedule and redeploy the App. A run can also be triggered manually from the dashboard.

Verify the workflow and serving endpoint

Check storage visibility

Use a harmless, uniquely named marker under /data rather than a production asset:

Write the marker inside a commit context from a write-capable Function.
End that invocation and run a reader that opens the file inside a reload context.
Compare the returned content with the expected marker.
Repeat the check with the read-only mount and confirm that a write attempt is rejected where your application expects rejection.
Exercise concurrent workloads only with disposable data; do not use production files to test the last-write-wins behavior.

Benchmark against the workload

The vLLM Benchmark CLI documentation lists ShareGPT as a supported dataset for online and offline benchmarks. Its documented download command is:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

For an online benchmark against a running vLLM server, the documented command form is:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

The generated report includes request throughput in requests per second, output-token throughput, and total-token throughput. The vLLM documentation positions this CLI for feature evaluation and regression testing. Keep the model, dataset, endpoint, GPU request, and application settings consistent when comparing runs, and record the generated results rather than treating one run as a capacity guarantee.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.