AI DeploySelf-hosted LLMGet it deployed

Automate LLM Evaluations on a Modal Schedule

Use a deployed Modal cron workflow to store LLM evaluation results and route recurring benchmark reports through a service you control.

Use a deployed Modal App with modal.Cron to run an LLM evaluation on a fixed cadence, attach a Modal Volume for reusable inputs and run artifacts, and pass the completed result to your own recurring-report adapter. Modal’s scheduling guide says deployment activates the schedule, modal.Cron remains fixed across redeployments while modal.Period resets, a timezone can be supplied, schedules cannot be paused, and the dashboard can trigger a run manually. For recurring ai deployment, define the evaluator, mount storage, deploy, check one manual run, and only then rely on the schedule.

Give ai deployment a fixed cadence

Start with the documented decorator fragment and retain the cron expression that matches your intended cadence:

@app.function(schedule=modal.Cron("0 6 * * *", timezone="America/New_York"))

The explicit timezone argument keeps the schedule definition attached to the function rather than relying on an unspecified local timezone. After implementing the function, deploy the App. The exact deployment form in Modal’s guide is:

modal deploy --name daily_heavy heavy.py

Replace heavy.py with the file containing your App and daily_heavy with your chosen App name. Editing a schedule locally does not activate it until you deploy that change.

A scheduled trigger also does not mean that a container remains permanently warm. Modal’s scaling documentation says every Function has its own autoscaling container pool and scales to zero by default when there are no inputs. Use the manual dashboard trigger to check scheduling, artifact creation, and report delivery first; do not treat that operational check as a performance benchmark.

Prepare a Volume for inputs and artifacts

Modal’s Volumes guide describes a Volume as a distributed file system optimized for write-once, read-many workloads such as model weights. It documents CLI creation and the v2 selector:

modal volume create
modal volume create --version=2

--version=2 creates a v2 Volume. The same guide marks v2 as Beta and says Modal does not recommend it for mission-critical data, so choose the version according to the current documentation and the importance of the evaluation artifacts.

You can also create the Volume lazily from Python and attach it through a mount path:

vol = modal.Volume.from_name("my-volume", create_if_missing=True)

@app.function(volumes={"/data": vol})

Use /data as the application’s mount path, but organize the directory by model revision, dataset revision, and run identifier. Put immutable fixtures or weights behind a read-only mount when appropriate; the documented form is:

volumes={"/data": volume.with_mount_options(read_only=True)}

Keep the result mount writable. Changes made inside a container must be committed before they become visible outside it. Modal also documents background commits every few seconds while a Function or Sandbox is running, followed by a final commit when the container shuts down. A container that mounted the Volume earlier must call .reload() to see later commits. Calling .reload() while files remain open can fail with volume busy.

Do not let concurrent runs update the same result file. Modal documents last-write-wins behavior for concurrent changes to one file and does not support distributed file locking. A separate artifact path for each run avoids that collision. Also, Volumes do not have a fixed capacity, so tools based on filesystem capacity such as df return placeholder values rather than a meaningful storage limit.

Configure the evaluator and its compute

If the evaluation container needs a GPU, attach one through the Function’s gpu argument. Modal states that GPU use requires a valid payment method. Its GPU documentation lists L40S as a valid value and recommends starting with that option for inference workloads; the documented configuration provides 48 GB of GPU memory, but you still need to verify that it fits your specific model and workload.

For reproducible measurements, record the requested GPU identifier rather than only the model name. Modal documents that a request for gpu="H100" may run on an H200, while gpu="H100!" opts out of that automatic change.

Merge the documented scheduling, storage, and GPU arguments into your evaluator’s decorator:

@app.function(
    schedule=modal.Cron("0 6 * * *", timezone="America/New_York"),
    volumes={"/data": vol},
    gpu="L40S",
)
def scheduled_evaluation():
    pass

This is only a framework scaffold. Replace pass with application code that loads the intended model revision, executes your quality checks and benchmark harness, writes a structured run artifact beneath /data, and invokes your report adapter. Do not deploy the placeholder as a working evaluator.

Separate quality checks from benchmark claims

An application-specific correctness check and a performance benchmark answer different questions. For correctness, version the prompts, expected outcomes, scoring function, dataset, and failure policy yourself. Do not describe those internal results as MLPerf results unless the evaluation follows the applicable MLPerf rules.

The MLPerf Inference rules require every submission to use LoadGen and define the system under test as the hardware and software resources being measured. In the Server scenario, LoadGen sends queries according to a Poisson distribution and the reported result is the maximum supported Poisson throughput under the scenario’s latency constraint. Offline sends all samples at the start and reports measured throughput without a latency constraint. Single Stream sends the next query after the previous response and reports a 90th-percentile latency estimate.

For LLM benchmarks, the same rules collect time to first token, or TTFT, and time per output token, or TPOT. TTFT covers the first generated token, while TPOT measures the average interval between generated tokens. Server latency constraints are expressed as category-specific TTFT/TPOT pairs. Do not compare results produced with different scenarios, system definitions, model revisions, or metric rules as though they were interchangeable.

Each run record should identify the benchmark and scenario, system under test, requested GPU, model and code revisions, dataset revision, metric settings, and Volume artifact path. Keep application-quality scores separate from TTFT, TPOT, and throughput fields.

Store, report, and verify one run

Have the evaluator create one artifact for the run, then build the recurring report from that same result rather than from unrelated manual observations. The report should identify the run and include the benchmark method, quality results, performance metrics, and artifact location. Its destination should be a reporting service you already operate.

The cited Modal documentation does not define an email, messaging, or report-delivery endpoint. Select your provider separately, follow that provider’s current documentation for its endpoint, authentication, retry behavior, and delivery status, and keep credentials out of source code.

Perform the first verification in this order:

  1. Deploy the App and use the manual trigger described in the scheduling guide.
  2. Confirm that your report provider received the summary for that run.
  3. Check the run artifact through the mounted Volume from another consumer, calling .reload() where required.
  4. Compare the report’s metrics and identifiers with the artifact contents.
  5. Confirm that the scheduled invocation produces the same artifact and report structure.

Modal’s Volume dashboard downloads support files up to 16 MB. For a larger artifact, the documentation directs you to the CLI download path named modal volume get; use the current syntax shown there rather than assuming that the dashboard can retrieve it.

Monitor only endpoints you control

If your own report adapter exports Prometheus metrics, configure a scrape target that actually exists. The Prometheus getting-started guide places targets under scrape_configs and shows a default global scrape interval of 15 seconds:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: YOUR_JOB_NAME
    static_configs:
      - targets: ['YOUR_TARGET']

Replace both placeholders with values from your instrumentation. The documented startup command is:

./prometheus --config.file=prometheus.yml

Prometheus does not discover a Modal evaluation endpoint merely because a schedule exists. Configure a target only if your application exposes it, and make sure the metrics distinguish successful runs, failed runs, artifact creation, and report delivery. Otherwise, a healthy Prometheus process can coexist with a broken evaluation workflow.

Common failure points

  • The schedule never runs: confirm that the App containing the scheduled Function was deployed after the schedule was added.
  • The timing moves after deployment: use modal.Cron for a fixed cadence rather than relying on the redeployment-reset behavior of modal.Period.
  • A result appears stale: account for commit timing, call .reload() when another container needs later changes, and close files before reloading.
  • One run overwrites another: use a distinct artifact path for each run because same-file concurrent writes are last-write-wins.
  • Results look comparable but are not: align the benchmark scenario, metric definitions, model revision, system under test, and input data.
  • The report or monitor stays empty: verify the report provider’s delivery record and confirm that the monitoring target is an endpoint your own code exposes.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use; nothing here was run or measured for this article.

Sources