Run Scheduled LLM Evaluations on Modal with Cron

Run an LLM evaluation on a fixed schedule by attaching modal.Cron to a Modal Function, deploying the App, and keeping reusable data in project-controlled storage; Modal’s scheduling guide covers the schedule and deployment path. For ai deployment, separate runtime orchestration from evaluation logic, record every attempt, and make the reviewer’s next action explicit.

Before ai deployment, choose what must persist

Separate stable inputs, such as model weights and evaluation datasets, from records produced by scheduled runs. Decide which data must be shared before writing the function body; a local path inside one container is not necessarily the location your reviewer or another container will use.

The Modal Volume guide describes Volumes as distributed filesystems optimized for write-once, read-many workloads such as model weights. It also documents lazy creation with create_if_missing=True and attachment through a function decorator:

vol = modal.Volume.from_name("my-volume", create_if_missing=True)

@app.function(volumes={"/data": vol})

These are separate documentation fragments, not two decorators to stack. Combine the Volume mapping with your function’s other supported arguments in the final function definition.

The same Volume guide says changes made inside a container must be committed before they become visible outside it, while another container must call .reload() to see later commits. For review, dashboard downloads support files only up to 16 MB; larger files require the CLI command modal volume get.

The correct model-upload and dataset-preparation commands depend on the selected artifacts, so use their current project documentation rather than copying a generic path. In this design, /data remains writable because the function both reads shared inputs and writes run records beneath that mount.

Build and deploy the fixed-time schedule

Keep the schedule definition in the same App file as the evaluation function. The Modal Cron SDK reference accepts a timezone argument and uses UTC when none is supplied, so set the intended timezone explicitly. Its documented constructor is modal.Cron(cron_string, timezone="UTC").

With app and vol already defined, the Modal-facing boundary can look like this:

@app.function(
    schedule=modal.Cron("0 6 * * *", timezone="America/New_York"),
    volumes={"/data": vol},
)
def run_evaluation():
    pass

The schedule, timezone, and Volume arguments are documented Modal settings. The function name and body belong to your project. The pass statement is deliberately only a boundary marker: replace it with the model-loading, dataset, endpoint, and evaluator calls required by your own stack before deployment.

After replacing pass, save the file as heavy.py. The cron guide says modal.Cron keeps fixed timing across redeployments, modal.Period resets its interval after redeployment, the schedule becomes active only after deployment, and a run can be triggered manually from the dashboard. Its deployment command is:

modal deploy --name daily_heavy heavy.py

Use the manual run to exercise the function and its result-writing path before relying on the next scheduled invocation. A manual execution check confirms orchestration and storage behavior, but it is not automatically a benchmark result.

Store results and make failures reviewable

Define the reviewer’s record before implementing the evaluator. Keep it compact enough for routine inspection and include an identifier, status, metrics, and failure information:

{
  "run_id": "YOUR_VALUE",
  "status": "YOUR_VALUE",
  "metrics": {},
  "error": null
}

Use a unique run identifier so a later execution does not obscure an earlier one. Set status to passed only after the evaluator finishes and its result is persisted. Use failed when the run cannot complete, and replace error with a sanitized explanation that helps the reviewer identify the next action without exposing credentials or unrelated sensitive data.

Keep operational failures separate from evaluation findings. An evaluator exception is an operational failure; an unacceptable model score is a completed evaluation with an unfavorable result; missing benchmark metadata is a validity problem. Mixing those states makes both incident review and model comparison harder.

This design surfaces failures through a stored status record rather than assuming that scheduling itself provides an alerting channel. If persistence fails, the Volume record cannot report that failure; add a separate alert path only after checking the provider’s current documentation.

Decide what the evaluation proves

First decide whether you are running a project-specific quality evaluation or an MLPerf Inference benchmark. Scheduling a quality evaluator does not make its score an MLPerf result.

For an MLPerf submission, the MLPerf Inference Rules require every submission to use LoadGen. They also define the system under test as the hardware and software resources being measured, so record enough environment information to reproduce that system.

For LLM benchmarks, the MLPerf rules collect time to first token and time per output token. The Server scenario sends queries according to a Poisson distribution and reports the maximum Poisson throughput supported under its latency constraint; Offline sends all samples at the start and reports measured throughput; Single stream sends the next query after the previous one completes and reports a 90th-percentile early-stopping latency estimate.

Before scheduling recurring runs, record the model revision, container image, libraries, hardware target, dataset, prompts, output settings, and selected scenario. Keep those inputs stable when comparing scheduled results. Never place metrics from different scenarios in the same comparison column, and label a project-specific quality score as custom rather than MLPerf.

Control GPU and cold-start effects

The GPU acceleration guide says a GPU is attached through the function decorator’s gpu argument and that using one requires a valid payment method. For a benchmark that must remain on H100 rather than accept the documented automatic H200 upgrade, request the explicit form:

gpu="H100!"

Use that value only when H100 is the intended benchmark target. Otherwise, select from the GPU guide’s current documented identifiers instead of inferring hardware from a model name. Record the target with the results so a later reviewer can distinguish an evaluation change from an environment change.

Modal’s cold-start guide defines a cold start as occurring when no ready container exists and Modal must start a new one. It identifies queue waiting and first-invocation initialization as additional latency sources, and recommends downloading model weights ahead of time instead of during container startup. Keep startup waiting outside model-quality or token-latency metrics, and pre-stage reusable weights before the scheduled workload depends on them.

Verify the run and troubleshoot common misses

After replacing pass, exercise the function manually and inspect the result record. A successful verification should answer these questions:

• Does the deployed file contain the schedule, explicit timezone, Volume mapping, and real evaluation body?
• Does a new run record appear beneath /data with the expected identifier?
• Does status distinguish completion from failure, with metrics and error information handled consistently?
• Are the model, dataset, scenario, and system-under-test details recorded when the evaluation claims benchmark validity?
• Are cold-start delays separated from model execution measurements?

Common misses occur at the boundaries. The schedule may exist in a file that was not deployed, or the function may still contain pass. The evaluator may run without writing beneath the expected mount, or a reviewer may inspect a record without following the commit and reload sequence described earlier. A result may also be overwritten because every run uses the same output record.

Evaluation failures often come from a different boundary: a custom score may be presented as MLPerf, scenarios may be mixed, model or software settings may change between runs, or startup waiting may be counted as model latency. Correct the orchestration, persistence, or measurement contract before drawing a conclusion from the number.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.

Sources

• Scheduling remote cron jobs
• Modal Cron Python SDK reference
• Volumes
• GPU acceleration
• Cold start performance
• MLPerf Inference Rules