Choosing Modal or Replicate for Python LLM APIs

Compare Modal GPU Functions with Replicate predictions, billing, API limits, and cold-start controls for a Python LLM deployment.

For ai deployment behind a Python LLM API, choose Modal when direct GPU and Function-level capacity controls matter; choose Replicate when a prediction lifecycle and model- or deployment-specific billing fit your Python client. This is a control-and-cost decision, not a speed guarantee. Match the documented workflow to your model, request pattern, and tolerance for queueing and cold starts.

How ai deployment priorities change the Modal–Replicate choice

Start with the boundary you want to own. Modal fits when accelerator type, GPU count, fallback order, and warm capacity are application decisions. Replicate fits when you call a model or deployment through prediction creation and polling, and its billing category matches the way you operate.

Bring request shape, not just model size. Decide whether calls may queue, whether a cold call is acceptable, and whether your client should wait for one response. Then collect current rates and billing terms for the exact GPU or model. If those inputs are not available from a provider page or quote, the comparison is incomplete.

What to prepare

Record the model version, runtime or container requirements, input and output schemas, memory needs, and intended concurrency. Decide whether Python will consume asynchronous results or wait for a response, and define timeout and cancellation behavior before using any example setting in production.

Identify the exact Replicate model class or deployment and read its “Run time and cost” information. For Modal, record the intended GPU request, fallback order, and capacity bounds. Put provider prices and usage values in the calculation only after taking them from the current page, bill, or quote.

Configure Modal from GPU to capacity

According to Modal’s GPU acceleration guide, GPU use requires a valid payment method and attaches hardware through a Function decorator’s gpu argument. The documented choices include T4, L4, A10, L40S, A100 variants, RTX-PRO-6000, H100/H100!, H200, B200/B200+, and B300; append :n for multiple GPUs, with most types supporting up to 8 and A10 up to 4. All GPUs in one container sit on one physical machine, requests for more than 2 GPUs usually wait longer, an ordered list provides fallback, and Modal recommends L40S with 48 GB of GPU memory as an inference starting point.

The documented fallback form is:

@app.function(gpu=["H100", "A100-40GB:2"])

Modal tries H100 first before considering the fallback. Treat this as a configuration example, not proof that either GPU fits your model.

Set Function capacity

Modal’s Scaling out guide says each Function has its own autoscaling pool, scales to zero by default without inputs, adds containers when capacity is unavailable, and removes idle ones. It defines max_containers as the upper limit, min_containers as warm inactive capacity, buffer_containers as spare active capacity intended to avoid queueing, and scaledown_window as the maximum idle duration in seconds before scale-down; changes through Function.update_autoscaler() revert to decorator values at the next deploy.

For a deployed Function, the documented call is:

f.update_autoscaler(min_containers=2, max_containers=10)

Do not copy those values as a preset. Set them from your cold-start tolerance, capacity policy, and traffic pattern.

Build the Replicate prediction flow

Replicate’s Create a prediction guide defines predictions.create for community models, models.predictions.create for official models, and deployments.predictions.create for deployments. Asynchronous creation is the default and returns a prediction ID with an incomplete object; Prefer: wait holds the request for 60 seconds by default, while Prefer: wait=5 waits 5 seconds. An unfinished result uses starting or processing; poll with GET requests to urls.get or the Location URL, with the documentation suggesting 1–2 seconds between checks. Cancel-After sets an automatic deadline from 5 seconds to 24 hours.

Choose the creation call from the target:

predictions.create
models.predictions.create
deployments.predictions.create

For a short synchronous wait, use the documented header form:

Prefer: wait=5

If the response remains incomplete, preserve it and poll until the status is succeeded or failed. The end of the wait header is not proof that inference finished.

Respect the request limits

Replicate’s rate-limit documentation sets prediction creation at 600 requests per minute and other endpoints at 3000 requests per minute, permits short bursts above the defaults, and returns HTTP 429 with reset information when throttled. It applies stronger limits as credit is nearly exhausted; granted-credit accounts without a payment method are limited to 1 request per second and at most 6 requests per minute.

Queue excess work, follow the reset information, and confirm that the account uses the billing arrangement you intended.

Verify the workflow you deployed

For Modal, invoke the entry point defined by your application. The cited guides do not provide a universal LLM health URL or response schema, so take the endpoint and payload contract from your deployed code. Confirm the intended gpu and autoscaler values, send a representative input, and repeat after an idle period if cold behavior matters.

For Replicate, confirm that creation returned a prediction ID and urls.get, then GET that URL. Continue while the state is starting or processing, and accept the output only after succeeded. HTTP request completion is not the application-level success check.

Calculate Replicate cost before choosing it

Replicate’s billing guide says public models charge only active processing time while setup and idle time are free; most private models and deployments charge for setup, idle, and active online time; fast-booting fine-tunes charge only active time; public models use a shared hardware pool and queue, while most private models use dedicated hardware without that queue; and a model that calls other models adds downstream charges. Failed runs are not charged, although a cancelled official-model run may still be charged; time-billed cancellation is charged through cancellation; and failed or cancelled private-model and deployment runs are billed for active instance time. Accounts in arrears are charged at the start of the month for the previous month’s usage, prepaid usage is deducted from the balance, and Replicate may charge early when first-time usage crosses certain limits.

Use the model page’s “Run time and cost” section to capture the applicable hardware and rate. Fill these placeholders from your own quote or invoice:

BILLABLE_TOTAL = Σ(BILLABLE_QUANTITY_i × QUOTED_RATE_i)
PUBLIC_ACTIVE_ESTIMATE = ACTIVE_PROCESSING_TIME × QUOTED_ACTIVE_RATE
PRIVATE_OR_DEPLOYMENT_ESTIMATE = (SETUP_TIME + IDLE_TIME + ACTIVE_TIME) × QUOTED_ONLINE_RATE
CHAINED_MODEL_ESTIMATE = ROOT_MODEL_CHARGE + DOWNSTREAM_MODEL_CHARGES

Normalize every time to the unit used by the quote. Apply the active-time estimate only when active processing is the billable category, and use the private or deployment formula only if one rate applies across all online phases; otherwise use the line-item formula. The Modal documentation used here does not settle a general Python LLM inference price formula, so obtain a current Modal quote and apply the same method rather than inferring cost from a GPU name.

Reduce cold starts and prevent billing surprises

Modal defines a cold start as creating a container when no ready container exists. Its cold-start guide says containers boot in about one second, but global-scope work such as imports and modal.enter methods must finish before the container is warm; it also recommends downloading model weights ahead of time as the first fix for slow boots. Queueing for a warm container and first-invocation initialization are separate parts of the delay.

On Replicate, treat the public or private model choice as part of both queueing and billing policy. Do not promise a latency outcome from the model class alone; check the current provider terms and validate the request pattern you actually need to serve.

Where the workflows usually go wrong

Copying the Modal example bounds instead of setting capacity policy.
Treating the GPU fallback list as unordered or ignoring shared physical placement.
Choosing a Replicate creation call before identifying the model or deployment.
Treating Prefer: wait as completion instead of polling.
Retrying HTTP 429 without following reset information or checking credit.
Applying active-time-only billing to private models or deployments.

The steps in this article are compiled from the official documentation linked in the text; no commands or workloads were run for it, so check the current versions before use.