AI DeploySelf-hosted LLMGet it deployed

vLLM vs Replicate: Self-Hosted vs Managed LLM Hosting

Compare self-hosted vLLM with managed Replicate hosting, including control, deployment steps, billing rules, API limits, and verification.

Self-hosted vLLM gives your team direct control of the model server, but it also makes the server, capacity, and validation your responsibility. Managed Replicate hosting shifts model-server administration away from your team. For llm hosting, the meaningful comparison is how much operational work you retain and how each billing model changes as usage starts, idles, repeats, or is cancelled.

What the llm hosting decision actually changes

The practical distinction is where runtime responsibility and cost calculation sit.

Decision area Self-hosted vLLM Managed Replicate
Runtime boundary Your team owns the host, process lifecycle, capacity, and endpoint exposure. Your application follows a prediction lifecycle that Replicate's create-prediction guide describes as asynchronous by default.
Cost and queues You calculate the total from your own infrastructure bills or supplier quotes. Replicate's billing guide says public models share a hardware pool and queue, while most private models use dedicated hardware without a shared queue; billing also differs by model class.
Main operating task Start, secure, observe, update, and validate the server yourself. Create predictions, track their states, and account for provider-specific scaling and billing.

Control does not eliminate work; it assigns more lifecycle decisions to your team. The managed path removes server administration but introduces billing classes, prediction states, and account limits that your application must handle.

Prepare and install vLLM

Start with the documented setup path: Linux and Python 3.10 to 3.13 are required. It creates a Python 3.12 virtual environment with uv, whose --torch-backend=auto option can inspect the installed CUDA driver and select the appropriate PyTorch index.

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

For NVIDIA GPUs, the GPU installation guide requires compute capability 7.5 or higher. It says the prebuilt vLLM package contains precompiled C++ and CUDA 12.9 binaries, while NVIDIA Blackwell GPUs such as B200 and GB200 require at least CUDA 12.8.

Use the isolated environment as the installation baseline. If package resolution becomes difficult, check the operating system, Python, GPU capability, and CUDA combination before combining it with unrelated Python dependencies.

Start and verify the self-hosted server

The documented server workflow hosts one model at a time. It listens on http://localhost:8000 by default, accepts --host and --port for the address, and checks API keys when --api-key or VLLM_API_KEY is configured. Several keys can follow --api-key, allowing rotation.

Start the documented example model with one command:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

Then request the model list from the documented endpoint:

curl http://localhost:8000/v1/models

A successful response confirms that the HTTP listing route is reachable. It does not, by itself, prove that an application's prompts, output format, authentication, or generation behavior is correct; those checks belong in a separate application-level validation step.

Use Replicate's managed prediction workflow

Replicate provides separate creation paths for community models, official models, and deployments. Its prediction API flow is asynchronous by default: the initial response returns immediately with a prediction ID and an incomplete prediction object.

For a synchronous wait, use the documented header example:

Prefer: wait=5

The default wait duration is 60 seconds. If the model has not finished when that duration ends, the response remains incomplete with status starting or processing. Retrieve the prediction URL from urls.get or the Location header and request it again; the documentation's polling example waits between 1 and 2 seconds between checks.

A Cancel-After header sets an automatic cancellation deadline. Its valid range is 5 seconds to 24 hours. Choose it according to the longest run your application can accept, and continue polling until the prediction reaches a terminal state rather than treating the initial asynchronous object as completion.

Calculate Replicate and self-hosted costs

Before calculating a Replicate total, use the billing documentation as a checklist:

  • Read the hardware specification under Run time and cost on the model page.
  • Identify whether the workload is a public model, a private model, a fast-booting fine-tune, or a deployment.
  • Record setup, idle, active, and downstream-model usage where it applies.
  • Check whether the account uses prepaid credit or monthly billing in arrears.
  • Record failed and cancelled-run treatment separately from successful usage.

Replicate documents several different billing paths:

  • Public models are charged for active request-processing time; setup and idle time are free.
  • Most private models are charged while instances are online, including setup, idle, and active time.
  • Fast-booting fine-tunes are charged only for active time, whether public or private.
  • Deployments are charged for setup, idle, and active time while instances are online.
  • If no model has demand, instances scale down to the configured minimum, which is 0 by default. After a request gap, an instance remains idle for a few minutes rather than shutting down immediately.
  • A model that calls other models adds downstream-model compute charges to its own compute time.

For a self-hosted estimate, fill the categories that actually appear in your own bill or quote:

SELF_HOSTED_TOTAL = YOUR_GPU_OR_SERVER_COST + YOUR_STORAGE_COST + YOUR_NETWORK_COST + YOUR_HOSTING_OR_POWER_COST + YOUR_OPERATIONS_COST

For managed usage, select the formula matching the documented billing class:

PUBLIC_TOTAL = YOUR_ACTIVE_PROCESSING_CHARGE + YOUR_DOWNSTREAM_MODEL_CHARGES

MOST_PRIVATE_OR_DEPLOYMENT_TOTAL = YOUR_SETUP_CHARGE + YOUR_IDLE_CHARGE + YOUR_ACTIVE_CHARGE + YOUR_DOWNSTREAM_MODEL_CHARGES

FAST_BOOTING_FINE_TUNE_TOTAL = YOUR_ACTIVE_PROCESSING_CHARGE + YOUR_DOWNSTREAM_MODEL_CHARGES

Omit the downstream term when the model does not call other models. Failed runs are not charged, but the exceptions matter: a cancelled official-model run may still be charged, time-billed runs are charged until cancellation, and failed or cancelled private-model and deployment runs are billed for active instance time.

Accounts billed in arrears are charged at the beginning of the month for the previous month's usage, while prepaid usage is deducted from the credit balance. Replicate may also charge early when usage first crosses certain limits as a fraud check.

Check Replicate API limits

Replicate's rate-limit documentation allows 600 prediction-creation requests per minute and 3000 requests per minute for all other endpoints. Short bursts above the default limits can be allowed before throttling. A throttled request receives HTTP 429 and a response body indicating when the limit resets.

Stronger limits apply as an account approaches running out of credit. Accounts granted credit but without a payment method are restricted to 1 request per second and a maximum of 6 requests per minute. These are API-control limits, not guaranteed throughput figures for a particular model or hardware configuration.

Validate performance without comparing unlike reports

The vLLM Benchmark CLI documentation positions vllm bench for feature evaluation and regression testing, and recommends GuideLLM for production-server benchmarking. ShareGPT is a supported dataset for online and offline benchmarks and is downloaded from Hugging Face.

For a server hosting the model named in this documented example, the online benchmark command is:

vllm bench serve \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt

Do not point that command at the earlier Qwen example unless the running server hosts the matching model. The report includes request throughput and output-token and total-token throughput. It also measures TTFT from request submission to the first streamed output, ITL between consecutive streamed outputs, and TPOT once per request while excluding the first token.

Metric names are not standardised across benchmarking tools. Repeating the benchmark against the same server may also reuse prompts retained in its prefix cache and inflate throughput, so record server state and compare like-for-like request patterns.

Where it usually goes wrong

  • Installation: Verify the documented OS, Python, GPU, and CUDA combination before debugging application settings.
  • Connectivity: Check the running process, bind address, expected model, and API-key configuration before changing client code.
  • Managed requests: Follow the prediction URL to completion instead of treating the first asynchronous response as the result.
  • Cost comparison: Classify the model and cancellation behaviour before adding quoted amounts; public, private, fine-tune, and deployment totals are not interchangeable.
  • Performance comparison: Align request arrival, concurrency, dataset, server state, and metric definitions before drawing conclusions.

Nothing here was run or measured. These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources