Configure Storage for a Self-Hosted vLLM Server

Guide vLLM weights, compile caches, and process logs across local disks, S3, containers, and persistent cloud volumes with clear lifecycles.

For a self hosted llm, separate model weights, the vLLM compile cache, and runtime files rather than assuming every writable path has the same lifecycle. vLLM persists torch.compile artifacts under VLLM_CACHE_ROOT, whose documented default is ~/.cache/vllm; the directory can be copied between machines or baked into a container image, according to the vLLM optimization guide.

Prepare storage for a self hosted llm

Write the following into the deployment configuration before launching the server:

Model source: Record the exact local checkpoint directory or s3:// object URI that the loader should read.
Compile cache: Record VLLM_CACHE_ROOT, including whether the default path is acceptable or an override is required.
Runtime files and logs: Identify what must survive a process restart, machine restart, container replacement, or rented-GPU Pod deletion.

These are deployment records, not additional vLLM requirements. Separating them prevents a temporary working directory, model checkpoint, and retained cache from sharing an accidental retention policy.

For a multi-node cluster, the vLLM parallelism and scaling guide recommends downloading a Hugging Face model before starting vLLM and using the same model path on every node. Record that path and verify it from each node rather than assuming that identical machine configurations imply identical mounted storage.

Install vLLM and verify the API

The vLLM quickstart lists Linux and Python 3.10 through 3.13 as prerequisites. Its recommended setup creates a Python 3.12 virtual environment with uv:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Here, --torch-backend=auto lets uv inspect the installed CUDA driver version and select the appropriate PyTorch index automatically.

Start a baseline server with the documented model command:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The server listens at http://localhost:8000 by default. Use --host and --port when it must listen on another address or port. A server process hosts one model at a time, so storage planning and model selection belong to the same deployment decision.

Check the model endpoint from a machine that can reach the configured address:

curl http://localhost:8000/v1/models

A returned model listing confirms that the API process is serving now. It does not establish that the same checkpoint remains reachable after a restart, so repeat this check after the lifecycle event you need to support.

Check Docker GPU access

Before loading a model in Docker, install the host NVIDIA drivers, reboot, and install the NVIDIA Container Toolkit, following Docker’s GPU access guide. Its verification command is:

docker run -it --rm --gpus all ubuntu nvidia-smi

This checks whether the container can see the GPUs. It does not mount model weights or configure VLLM_CACHE_ROOT; pair it with the image, checkpoint mount, cache location, and runtime identity selected for your deployment.

Load weights from local or S3 storage

The Run:ai Model Streamer guide describes a loader that reads tensors concurrently while streaming them into GPU memory. If you choose this extension, install its optional dependency in the active environment:

pip3 install vllm[runai]

For a checkpoint already present on the server’s filesystem, the guide gives this local-path example:

vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \
  --load-format runai_streamer

For an object stored in S3, it documents serving directly from an s3:// URI:

vllm serve s3://core-llm/Llama-3-8b \
  --load-format runai_streamer

Treat both locations as documented examples. The command does not configure credentials, so use the authentication arrangement required by your S3 environment and check the current loader guide rather than adding an undocumented credential flag.

Loader concurrency and the CPU read buffer are controlled through --model-loader-extra-config. The guide provides these option examples:

--model-loader-extra-config '{"concurrency":16}'

--model-loader-extra-config '{"memory_limit":5368709120}'

They are examples, not universal deployment defaults. For tensor- or pipeline-parallel deployments, the guide also names runai_streamer_sharded; it says each worker can read its own shard instead of the entire checkpoint.

Keep the compile cache under control

Set VLLM_CACHE_ROOT in the environment that launches vLLM, then record the resolved directory in the service definition. Keep it separate from model weights and process logs.

If the cache will be copied between machines, configure the destination before startup and verify that the runtime identity can use it. If the cache will be baked into a container image, populate the directory before building the final image and retain the VLLM_CACHE_ROOT setting when that image starts.

Treat the cache as a separate operational artifact. It is not a replacement for the model checkpoint or a backup. Process output is different again: route standard output and standard error through the selected service manager or container runtime, then define log retention independently.

Match RunPod storage to its lifecycle

RunPod distinguishes container disk, volume disk, and network volume storage. The following matrix follows its storage options documentation.

Storage type	Persistence	Configuration consequence
Container disk	Created when the Pod launches and cleared when the Pod stops.	Do not depend on it for data that must survive a stop.
Volume disk	Data in `/workspace` survives stops and restarts but is deleted when the Pod is terminated.	Its size can be increased, but not decreased.
Network volume	Exists independently of a Pod and remains after the Pod is deleted. It can be attached to multiple Pods and transferred between machines.	It must be attached when the Pod is created, cannot be detached later, and replaces the volume disk at `/workspace`.

Encryption applies only to the volume disk; neither container disk nor network volumes can be encrypted. RunPod also states that its platform is not designed for long-term cloud storage and recommends backing up critical data to a local machine or a dedicated cloud storage provider.

Use that lifecycle distinction when placing model weights. A network volume can retain data after Pod deletion, but independent retention is not a substitute for an external backup. Keep another copy of irreplaceable weights according to your organization’s recovery requirements.

Common storage mistakes and final checks

The model source does not match the loader: Compare the active vllm serve command with the intended local path or S3 URI. A baseline model-ID command does not prove that a separately staged checkpoint is being used.
The cache root is missing from the deployment record: Record the exact VLLM_CACHE_ROOT and decide separately whether it must survive rebuilds, copies, or machine replacement.
Nodes use different model paths: Open the configured path from every node before startup. Matching model names do not guarantee matching storage paths.
GPU visibility is mistaken for a storage fault: Run the Docker GPU verification command before investigating model-loading errors in a container.
RunPod retention is assumed incorrectly: Choose container disk, /workspace, or a network volume according to whether data must survive a stop, termination, or Pod deletion. Remember that a network volume must be attached at Pod creation.
There is no external backup: Follow the provider’s warning and preserve critical weights outside the rented Pod environment.
Process output has no owner: Define where standard output and standard error are written, how long they are retained, and who monitors them.
Persistence is never checked: After /v1/models responds, repeat the check after the intended restart or recreation event.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.