AI DeploySelf-hosted LLMGet it deployed

Connect RunPod Networking to a Self-Hosted LLM

Connect a private LLM through RunPod internal networking or its public HTTP proxy, attach persistent storage, and verify the correct vLLM API.

A self hosted llm on RunPod can be connected through RunPod Global networking for Pod-to-Pod traffic or through RunPod’s public HTTP proxy for external clients. Choose the caller path before changing DNS or firewall rules, then make the inference service reachable on its internal port and enforce authentication inside every public application. Keep model files on storage selected for the required recovery window instead of treating a Pod as the backup boundary.

Connect a self hosted llm through RunPod

Start with the caller rather than the port:

  • Use private Pod networking when the model serves backend services that should not accept internet traffic.
  • Use the public HTTP proxy when approved clients connect from outside that private boundary.
  • If another account must connect, do not assume account-private addressing applies. Cross-account peering is not settled by the linked RunPod documentation, so verify its current networking documentation before designing around an undocumented hostname or command.

RunPod’s Global networking guide gives each Pod a private IP address reachable only from other Pods in the same account, with this documented address form:

abc123xyz.runpod.internal

Other Pods use that hostname instead of the public proxy URL, together with the model service’s internal port. The documented feature is limited to NVIDIA GPU Pods and listed data centers, so confirm that the selected Pod and region remain supported before building private service discovery around the hostname. The documentation does not identify a universally fastest data center; choose from current availability and your own operational requirements rather than assuming a location ranking.

Prepare storage before exposing the service

Start the inference service with the model-specific procedure in the current vLLM documentation. This networking guide begins once that service is listening on an internal HTTP port; it does not substitute an unverified model, quantization setting, or launch flag.

RunPod’s Storage options distinguishes the relevant lifetimes: the container disk is created at launch and cleared when the Pod stops; data under /workspace on the volume disk survives stops and restarts but is deleted when the Pod is terminated; and a network volume exists independently and retains its data after the Pod is deleted.

/workspace

The Network volumes guide says to select a data center, enter a name, and specify the size in GB when creating a volume. For Pods, the network volume is available only in Secure Cloud, must be attached during Pod creation, cannot be detached later, and replaces the volume disk at /workspace; inside Serverless workers, it mounts at:

/runpod-volume

Treat regional placement and concurrent access as operational constraints. The same network-volume documentation warns that attaching one network volume to a Serverless endpoint pins its workers to that volume’s data center, which can limit GPU availability and reduce failover options. Simultaneous writes from several workers can corrupt data, and storage charges can continue while the Pod is stopped; if the balance cannot cover them, the volume may eventually be terminated and its data cannot be recovered. Your application must therefore prevent concurrent writes to the same volume and keep an independent recovery copy of critical files.

Configure public HTTP ingress

With the inference service running, use the documented address and field names:

0.0.0.0

Expose HTTP Ports (Max 10)

https://[POD_ID]-[INTERNAL_PORT].proxy.runpod.net

Apply the RunPod Expose ports guide as one sequence: bind the service to 0.0.0.0 rather than localhost, enter a comma-separated list containing no more than the field’s documented maximum, and use the proxy hostname for the corresponding internal port; the proxied service is publicly reachable and must implement its own authentication, while its Cloudflare proxy closes a connection after 100 seconds without a response and returns a 524 error.

Configure authentication inside the model application before distributing the proxy URL. Do not treat the Pod ID as an access-control boundary or assume that an internal port becomes private merely because clients normally use the public proxy hostname. Also distinguish a service-binding problem from a timeout: an unreachable URL points back to the bind address and exposed-port entry, while a delayed response requires examining streaming behavior and both proxy timeout layers.

Preserve token streaming through nginx

Buffering can make a functioning token stream appear stuck. The nginx proxy module documentation states that proxy_buffering is on by default, while disabling it passes the response to the client synchronously as it arrives:

proxy_buffering off;

If you cannot change the nginx location configuration, the upstream application can request the same behavior through this response header:

X-Accel-Buffering: no

The nginx proxy module documentation also gives proxy_read_timeout a default of 60 seconds between successive read operations, not for the total duration of an entire response. A stream that continues sending chunks can therefore run longer than that interval, but a gap between reads can still trigger the timeout. Increasing an nginx timeout alone does not remove the separate RunPod proxy’s 100-second connection limit.

The linked RunPod and nginx pages do not settle certificate authorities, domain names, certificate renewal, or other TLS deployment details. Check the current nginx TLS documentation and RunPod ingress documentation for those settings rather than copying an unverified certificate configuration.

Verify the vLLM API paths

The vLLM OpenAI-Compatible Server documentation states that its HTTP server implements OpenAI’s Completions and Chat APIs. Start with the exact route that matches the model and application:

/v1/completions

/v1/chat/completions

/v1/responses

For a private caller, target the Pod’s internal hostname and configured service port. For an approved public caller, target the RunPod proxy URL, append the selected API path, and supply the authentication required by your application. A connection that reaches the intended API route and produces the expected response shape verifies routing, port exposure, and application authentication; it does not establish a performance benchmark.

The vLLM endpoint documentation also defines important request differences: /v1/chat/completions requires a chat template and ignores the user parameter, while /v1/completions does not support the suffix parameter. Check the model type before using /v1/completions or /v1/embeddings, because the former applies to text-generation models and the latter to embedding models. For a streaming request, verify that output begins reaching the client as it is produced rather than waiting for the complete response.

Troubleshoot by network layer

Work outward from the process instead of changing every setting at once:

  • Private hostname fails: revisit the supported Pod type, data center, account boundary, and internal port described in the Global networking section.
  • Public proxy cannot connect: compare the process bind address, the Expose HTTP Ports (Max 10) value, and the internal port embedded in the proxy URL.
  • Authentication fails: verify the authentication implemented by the application; proxy reachability does not replace application access control.
  • The first token arrives late: inspect proxy_buffering and any X-Accel-Buffering response header before changing model settings.
  • A connection ends with 524: follow the outer proxy timeout path rather than assuming the inference process never started.
  • Files are missing or inconsistent: compare the selected storage lifetime, Pod termination state, and whether multiple workers can write concurrently.
  • The route responds but behaves unexpectedly: confirm whether the application selected Completions, Chat, or Responses and review that endpoint’s documented parameters.

These steps are compiled from the official documentation linked in the text; no commands or services were run for this article, so check them against the current version before use.

Sources