AI DeploySelf-hosted LLMGet it deployed

Fine-Tune an LLM with LoRA on Replicate GPUs

A documented workflow for packaging a LoRA training function with Cog, assigning a Replicate destination model, and submitting the training request.

Fine-tune an LLM with LoRA on Replicate by implementing the training logic in train.py, exposing it through Cog’s training interface, and creating a training request through Replicate’s HTTP API. These platform documents define the container and request mechanics, but they do not prescribe one LoRA library, dataset format, GPU model, or adapter schema; those choices belong to the training project you select. For this ai model deployment workflow, make each project-specific choice explicit before submitting the job.

Prepare for ai model deployment on Replicate

Set up Cog’s container and platform prerequisites. Cog packages machine-learning models as standard containers that can run on your own infrastructure or on Replicate. It requires Docker; if you use Docker Engine rather than Docker Desktop, Buildx must also be installed. Cog does not run natively on Windows, although Windows 11 users can run it through WSL 2.

On macOS, the documented installation command is:

brew install replicate/tap/cog

Before editing the container configuration, prepare:

  • A LoRA training implementation and its callable entry point.
  • The exact base-model version and its owner, name, and version identifier.
  • An existing destination model that you control.
  • A defined input schema for datasets, configuration values, and any model-specific options.
  • A documented output format for the resulting adapter or other training artifact.
  • A target self-hosted runtime that explicitly supports that output format.

Fine-tune the LLM: declare the training entry point

The training-interface reference uses a train key in cog.yaml that points to a function or class. For a function-based project, the declaration is:

train: "train.py:train"

The string after the colon must match the callable exposed by train.py. If you rename or move that callable, update the configuration rather than relying on inference.

The same reference marks the cog train command as deprecated. Treat the command-line training workflow as legacy material and use Replicate’s HTTP training endpoint for the procedure below.

Implement LoRA inside train.py

Keep LoRA configuration separate from GPU configuration. The container tells Cog how to build and run the project; your train() implementation decides how the selected model is adapted and trained.

Before writing that function, settle the following against the chosen LoRA project’s official documentation:

  • Which base model and tokenizer revision to load.
  • Which library and target modules implement the adaptation.
  • How the dataset is read and preprocessed.
  • Which optimizer, schedule, precision, and checkpoint settings apply.
  • Whether the result is an adapter, a merged model, or another artifact.
  • Where the artifact is written and how its integrity is validated.

Do not invent a package-install command, dataset JSON shape, target-module list, or hyperparameter block when those details depend on an unspecified model and framework. An exact example would be misleading. Likewise, do not assume that an artifact called an “adapter” can be loaded by a particular deployment runtime; verify the required format, naming rules, base-model pairing, and loading procedure first.

Run the training on Replicate GPUs

For a GPU-enabled Cog image, the cog.yaml reference documents this structure:

build:
  gpu: true

train: "train.py:train"

With build.gpu: true, Cog uses the nvidia-docker base image and determines the CUDA and cuDNN versions from the Python, PyTorch, and TensorFlow versions in the project. That setting does not itself add LoRA behavior, select a particular Replicate GPU, establish available memory, or prove that the model will fit.

Treat GPU selection as a separate compatibility decision. Check the framework and model requirements against the CUDA and cuDNN versions selected by Cog, then confirm the current hardware offered for the model or training job. If your chosen framework requires different versions, follow the current Cog configuration reference rather than pinning values from an unrelated example.

Make the model version available

The documented authentication and registry naming sequence in Replicate’s push guide is:

cog login
cog push r8.im/YOUR_USERNAME/YOUR_MODEL_NAME

The username and model name must match the values configured for the model on Replicate. Resolve the identifiers first, then use those same values in the training request.

Be aware of a documentation mismatch: Replicate’s push guide still demonstrates predict.py, Predictor, and cog predict. Do not combine that older inference interface with the current training declaration train: "train.py:train". Use the guide for authentication and naming, while following the dedicated training reference for the callable and configuration.

Submit the Replicate training request

Create the training through the documented HTTP API:

POST https://api.replicate.com/v1/models/YOUR_MODEL_OWNER/YOUR_MODEL_NAME/versions/YOUR_VERSION_ID/trainings

The request has three important elements:

  • The URL identifies the model version being trained.
  • destination identifies an existing model owned by the requester.
  • input is an object containing the inputs for the Cog model’s train() function.

Do not guess the complete authentication header or request body. Take the current serialization, required fields, and authorization method from the HTTP reference, then map input to the exact parameter names accepted by your train() implementation.

Verify the setup without confusing submission with success

Before sending the request, check five boundaries:

  1. cog.yaml points to the callable that actually exists.
  2. The GPU build setting is compatible with the selected training framework.
  3. The model owner, model name, and version ID identify the intended source.
  4. The destination model exists and belongs to the requester.
  5. Every input name and type matches the training function’s contract.

After submission, do not treat an accepted request as proof that the model trained successfully or that the resulting adapter can be deployed. The linked Replicate and Cog pages do not define a universal LoRA artifact schema, quality threshold, download command, or self-hosting compatibility contract.

Use the selected training project’s documentation to validate the saved artifact. If you plan to load it into a self-hosted runtime, then verify the exact artifact format against that runtime’s official documentation before moving any weights or changing a production deployment.

Calculate the charge from current documentation

Replicate’s billing documentation distinguishes several billing categories. It says public models are charged for active processing time, most private models are charged while instances are online—including setup and idle time—and deployments are also charged for online time. It separately states that fast-booting fine-tunes are charged only for active processing time, whether public or private.

Do not assume which category applies to your custom Cog LoRA training request. Read the current model page, especially the “Run time and cost” section, and obtain a training-specific quote or identify the applicable line items in the Replicate dashboard. Then calculate:

YOUR_ESTIMATED_TRAINING_CHARGE =
  YOUR_BILLABLE_GPU_TIME × YOUR_QUOTED_GPU_RATE
  + YOUR_DOCUMENTED_STORAGE_CHARGE
  + YOUR_OTHER_DOCUMENTED_CHARGE

Make sure the time units in the first two values match. Add only charges identified in the current quote or account record; do not populate the formula from a public model’s runtime price.

Common failure points

  • Mixing old and current interfaces. A predict.py example from the push guide does not define the train.py:train entry point used by this workflow.
  • Using incompatible YAML keys. The cog.yaml reference says python_packages is deprecated in favor of python_requirements, and only one of the two may be set.
  • Skipping platform prerequisites. Cog’s platform guidance requires Buildx with Docker Engine and WSL 2 rather than native execution on Windows.
  • Equating build.gpu: true with a complete training setup. It configures the image environment; it does not supply LoRA settings, select a guaranteed GPU, or establish model fit.
  • Stopping verification at submission. Request acceptance, artifact generation, artifact quality, and runtime compatibility are separate checks.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources