Evaluate a Public Replicate Model for Production

Public Replicate billing covers active processing, while production checks must cover queues, API limits, timeouts, licensing, and failure charges.

For ai model deployment, evaluate a public Replicate model as a hosted runtime: verify its category, billing basis, queueing behavior, API limits, request lifecycle, and failure charges before approving it for production. Replicate’s billing documentation says public models are charged for active processing rather than setup or idle time, but the default shared hardware pool and queue can lead to cold boots or scaling limits. One successful example request is not enough; require a current cost method, a workload test, and a failure plan.

Production criteria for ai model deployment

Start with an evaluation record, not a successful request. Record:

Identity and rights: the exact model page, owner, version, model card, license, source code and weights, and any data-handling terms.
Workload: input shape and distribution, expected output, arrival pattern, concurrency, deadline, and failure policy.
Evidence: a current cost method, repeatable workload tests, task-accuracy and latency criteria, and logs that retain errors and throttling.
Access and fallback: who can change visibility, what alternative route exists, and how an unavailable endpoint will be handled.

Public availability does not itself establish ownership or commercial permission. The linked runtime pages do not settle those terms, so obtain the model and project license before approval. Set pass and fail conditions before collecting results; otherwise, easy requests can bias the choice.

Classify the model before estimating cost

Replicate’s prediction creation documentation separates community-model, official-model, and deployment creation:

Community models - predictions.create
Official models - models.predictions.create
Deployments - deployments.predictions.create

These categories are not interchangeable. Replicate’s official-model documentation says official models are always warm and are priced by output metrics—such as images, video seconds, and input or output tokens—rather than hardware and run time. They are called through the following endpoint without specifying a version:

POST /models/<owner>/<name>/predictions

Do not apply that versionless route to a non-official public listing merely because it is publicly accessible. Record the model’s actual API category before choosing a billing formula or writing the client.

Build the public-model cost worksheet

Open the model page and record the hardware listed under Run time and cost. Replicate says that this heading shows the hardware assigned to the model. Obtain current rates from the model’s present page, quote, or bill rather than copying an old estimate.

For a general public model, use a worksheet such as:

PUBLIC_MODEL_ESTIMATE =
  Σ(YOUR_ACTIVE_TIME_i × YOUR_CURRENT_ACTIVE_RATE_i)
  + Σ(YOUR_DOWNSTREAM_TIME_j × YOUR_DOWNSTREAM_RATE_j)

The downstream line matters because a model that calls other models is billed for both its own compute and the models it calls. For an official model, calculate from published output units instead:

OFFICIAL_MODEL_ESTIMATE =
  Σ(YOUR_OUTPUT_UNITS_k × YOUR_PUBLISHED_UNIT_PRICE_k)

Do not apply the public-model active-time formula mechanically to every Replicate mode. The billing guide also establishes several exceptions:

Public-model setup and idle time are free, while active processing is charged.
An undemanded model scales down to its configured minimum, which is 0 by default; custom minimums are described for deployments.
After a gap in requests, an instance can remain idle for a few minutes instead of shutting down immediately.
A failed public run is not charged. The guide separately warns that a cancelled official model may still be charged and that a cancelled time-billed run is charged until cancellation.
Private models and deployments are billed for online setup, idle, and active time; their failed or cancelled active time can also be billed.
Accounts billed in arrears are charged at the start of the month for the previous month’s usage, while prepaid usage is deducted from the balance. Replicate may also charge early when usage first crosses certain limits as a fraud check.

Enter all values from your own current quote or statement. Do not turn this worksheet into a cost claim until you have checked the model category, hardware, downstream calls, cancellation behavior, and account billing method.

Exercise the complete request lifecycle

The prediction creation reference makes asynchronous creation the API default. It returns immediately with a prediction ID and an incomplete prediction object, which means the request was accepted—not that the model finished.

Synchronous mode holds the request open for 60 seconds by default:

Prefer: wait

The wait duration can be changed. For example, this header waits for five seconds:

Prefer: wait=5

If the prediction does not finish within that duration, the returned object can have a status of starting or processing. Retrieve the prediction URL from urls.get or the Location header and poll it:

GET YOUR_PREDICTION_URL

The documentation’s example waits 1–2 seconds between checks. Treat succeeded and failed as completed states, and validate the returned output separately when the state is succeeded.

Also set a prediction deadline when the workload requires one:

Cancel-After: YOUR_DEADLINE

Replicate documents valid Cancel-After values from 5 seconds through 24 hours. The synchronous wait is therefore not a completion guarantee; your client still needs an overall deadline, cancellation handling, and a record of whether processing started.

Protect the API from throttling

Replicate’s rate-limit documentation sets the default limit for prediction creation at 600 requests per minute and all other endpoints at 3000 requests per minute. Short bursts above those defaults can be allowed before throttling, so they should not be treated as guaranteed burst capacity.

When a limit is reached, the documented response uses HTTP 429 and includes a reset hint such as:

{
  "detail": "Request was throttled. Your rate limit resets in ~30s."
}

Do not hard-code that example interval. Parse the reset information returned for the request, pause new work, and avoid synchronized retry storms. Account for both prediction creation and polling traffic in separate client-side limiters.

Two account conditions also matter. Replicate applies stronger rate limits as an account approaches running out of credit. An account with granted credit but no payment method is limited to 1 request per second, with a maximum of 6 requests per minute. Record the applicable account condition before running a load test, and retain throttled requests in the results rather than silently discarding them.

Verify behavior with a production-shaped benchmark

If you need an MLPerf result, follow the MLPerf Inference rules. Every submission must use LoadGen, which generates queries for each scenario, tracks latency, validates accuracy, and computes the final metrics. A system under test is a defined set of hardware and software resources being measured.

MLPerf defines general latency from the moment LoadGen passes a query to the system under test until it receives the reply. For LLM benchmarks, it collects two latency metrics:

Time to First Token: the latency of the first token.
Time per Output Token: the average interval between generated tokens.

Choose the scenario that matches the workload rather than mixing unlike results:

Server: queries follow a Poisson distribution, each scenario has a benchmark-specific latency constraint, and the result is the maximum Poisson throughput supported.
Offline: LoadGen sends all samples at the start in a single query, has no latency constraint, and reports measured throughput.
Single stream: the next query starts when the previous query completes, and the metric is a 90th-percentile early-stopping latency estimate.

For a custom production evaluation, create an explicit record:

YOUR_MODEL = YOUR_VALUE
YOUR_HARDWARE = YOUR_VALUE
YOUR_INPUT_SHAPE = YOUR_VALUE
YOUR_OUTPUT_TARGET = YOUR_VALUE
ARRIVAL_PATTERN = YOUR_VALUE
CONCURRENCY = YOUR_VALUE

Run representative, sanitized inputs through the same model and endpoint type you intend to operate. Record TTFT, TPOT where applicable, task accuracy, successful and failed predictions, throttles, and active processing time. Keep cold and warm observations separate when cold boots occur, and do not calculate results only from the requests that completed. If the procedure does not follow MLPerf rules, label it as a custom workload test rather than an MLPerf result.

Production acceptance: common failure points

Avoidable review errors include:

Treating every public model as an official model and applying the wrong endpoint or price basis.
Treating an asynchronous prediction ID as a completed result.
Hard-coding rate-limit reset behavior or ignoring the applicable account restriction.
Counting only root-model compute while omitting downstream calls.
Testing one convenient input and calling it representative.
Assuming public visibility supplies a license, service-level commitment, or durable access.

Replicate’s private and public model documentation says that changing your public model to private immediately cuts off everyone using it. Include that access change in the continuity plan. The linked pages do not settle model licensing, data-processing terms, or an SLA, so check the current model, project, account, and contract documents separately.

Approve the model only when the record contains verified rights, a current cost worksheet, a workload-shaped result against predefined thresholds, tested throttling and cancellation paths, and a fallback plan. Otherwise, keep the production decision open. This is general production-planning information, not legal advice.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use. Nothing here was run or measured for this article.