AI DeploySelf-hosted LLMGet it deployed

Configure Time-Sliced GPU Jobs on Modal

Configure concurrent GPU jobs on Modal with input concurrency or batching, while separating Modal scheduling from Kubernetes GPU time-slicing.

On Modal, the documented way to run overlapping short GPU jobs is concurrent inputs on one Function or dynamic batching; Modal’s guides do not define a literal GPU “time-slicing” flag for a Function. Literal GPU time-slicing is documented in NVIDIA’s Kubernetes GPU Operator guide, where replicas of one GPU can be assigned independently to pods, so it should not be equated with Modal input concurrency. For ai deployment, choose concurrency or batching deliberately, then align warm-container behavior with the job pattern.

Define the Scheduling Target

“Time-sliced” can refer to different scheduling models. Modal concurrency controls how many inputs a Function container processes, while batching waits for several requests before executing them. Kubernetes GPU time-slicing instead creates replicas of a GPU that can be assigned to different pods.

Under NVIDIA’s documented time-slicing model, GPU processes across the pods receive equal shares of GPU time. Replicas do not have memory or fault isolation between them, so this mechanism does not provide the isolation associated with hardware-partitioned GPU instances.

For this ai deployment pattern, record the execution requirement before editing the Function:

  • Use Modal input concurrency when independent inputs may run at the same time.
  • Use Modal dynamic batching when requests should wait for a group.
  • Use Kubernetes GPU time-slicing when separate pods need replicas of a physical GPU.

These are different scheduling contracts. Do not apply settings from one model as though they configure another.

Attach and Constrain the GPU

A GPU is attached to a Modal Function through the gpu argument on its function decorator, and Modal requires a valid payment method on file to use GPU resources. According to Modal’s GPU acceleration guide, the documented gpu values are T4, L4, A10, L40S, A100, A100-40GB, A100-80GB, RTX-PRO-6000, H100, H100!, H200, B200, B200+, and B300.

Selection caveats matter when writing this argument. The same Modal guide documents that most listed GPU types support up to eight GPUs per container, while A10 supports up to four; requests for more than two GPUs usually have longer waits, and all GPUs in one container are attached to the same physical machine. It also documents automatic H100-to-H200 and A100-to-80-GB-A100 upgrades unless the exact variant is requested, B200+ billing as B200 regardless of the assigned GPU, and a CUDA 13.1-or-later requirement for B300.

A single GPU request uses the documented gpu value directly:

@app.function(gpu="H100")

Append :n when requesting multiple GPUs in one container:

@app.function(gpu="H100:8")

For fallback, pass an ordered list. Modal tries the preferred type before moving to less-preferred entries:

@app.function(gpu=["H100", "A100-40GB:2"])

Keep the order intentional, and confirm that an automatic upgrade or fallback would still satisfy the job’s requirements.

Configure Concurrent GPU Inputs

The Modal input concurrency guide states that each container handles one input at a time by default. The @modal.concurrent decorator enables multiple inputs; max_inputs is the per-container ceiling, while target_inputs is the concurrency level the autoscaler aims for. Active containers may burst above the target up to max_inputs. When demand exceeds that ceiling, Modal adds containers, although additional inputs may queue while those containers cold start.

The documented decorator example is:

@modal.concurrent(max_inputs=100)

Use your own resource and latency values in the same settings:

@modal.concurrent(
    max_inputs=YOUR_VALUE,
    target_inputs=YOUR_VALUE,
)

Set target_inputs around the concurrency you want the autoscaler to pursue. Set max_inputs no higher than the GPU and memory resources can support; Modal specifically advises using resource constraints, including avoiding GPU out-of-memory failures, as the basis for that ceiling.

For a synchronous Function, concurrent inputs run on separate threads, so shared mutable state must be thread-safe. Audit caches, clients, buffers, and global objects before increasing concurrency. An asynchronous Function instead uses asyncio tasks on one thread.

Configure Dynamic Batching

Dynamic batching solves a different scheduling problem. Modal’s dynamic batching documentation says execution begins when the lesser of a fixed waiting time or a fixed request count is reached. The decorator accepts max_batch_size and wait_ms:

@modal.batched(max_batch_size=2, wait_ms=1000)

For your workload, replace the documented sample values with your own:

@modal.batched(
    max_batch_size=YOUR_VALUE,
    wait_ms=YOUR_VALUE,
)

Choose the count that fits the work your function can process as one batch. Choose the time limit according to how long an input may wait for other inputs. This mechanism groups logical requests; it does not turn a Modal GPU request into a Kubernetes GPU time-slicing configuration.

Account for Cold Starts and Idle Containers

A cold start occurs when Modal has no ready container and must start one. The Modal cold-start guide attributes the resulting delay to queueing for a warm container, first-invocation initialization, or both. Global-scope logic, imports, and modal.enter methods must finish before the container is considered warm. Moving initialization into modal.enter shifts the delay into warm-up rather than removing it; Modal recommends downloading model weights ahead of time and, where feasible, reading large files concurrently rather than sequentially.

Three autoscaling settings control warm capacity: min_containers sets the minimum kept warm, buffer_containers maintains extra capacity while the Function is active, and scaledown_window sets the maximum idle duration in seconds before scale-down. According to the same cold-start guidance, the default maximum idle time is 60 seconds, and scaledown_window can be set from 2 seconds to 20 minutes. Increasing it can reduce later cold starts, but resources used by an idle container—including GPU reservation or residual memory occupancy—remain billable.

Submit and Verify Background Jobs

For background jobs, Modal’s job-processing guide documents deploying the Function, submitting work with modal.Function.spawn(), and retrieving the result with modal.FunctionCall.get():

deploy the Function
modal.Function.spawn()
modal.FunctionCall.get()

Results from .spawn() remain accessible through FunctionCall.get() for up to seven days after completion. The same Modal job-processing documentation states that the job queue can hold up to one million tasks; treat that as a documented queue capacity, not as a throughput or completion-time promise.

Configuration and Failure Checklist

Before deployment, verify the following in your own environment:

  • Wrong mechanism: Confirm that the requirement is Function-level concurrency or batching, not pod-level Kubernetes GPU replicas.
  • GPU ambiguity: Review fallback order, automatic upgrade behavior, multi-GPU count, and any B300 CUDA requirement.
  • Concurrency ceiling: Set max_inputs against available GPU memory, set target_inputs from your latency objective, and make synchronous code thread-safe.
  • Batch thresholds: Confirm the function can process the selected batch and that wait_ms matches the delay your callers can tolerate.
  • Warm-up delay: Download weights ahead of startup and review global initialization before treating an idle Function as immediately ready.
  • Result retrieval: Submit through the documented spawn-and-get path and retrieve results before their documented availability window closes.

For verification, deploy the Function, submit a representative job through .spawn(), retrieve it through FunctionCall.get(), and check the returned result against your expected output. Check configuration and behavior separately: correct output does not by itself prove that concurrency, batching, fallback order, or warm-container behavior is the intended configuration.

This guide is compiled from the official documentation linked in the text; no commands or workloads were run, so check the current version before use.

Sources