Run Concurrent LLM Jobs with Modal Functions

Use Modal Functions, GPU settings, and parallel calls to run concurrent LLM inference jobs and collect ordered results in one workflow.

When you need llm hosting on rented GPUs, define each inference job as a Modal Function, attach a GPU through the function decorator’s gpu argument, and submit independent inputs with Function.map(), following Modal’s GPU documentation. Modal runs mapped inputs in parallel and returns their results in input order; each Function also has its own autoscaling container pool that scales to zero by default when no inputs remain, as described in Modal’s scaling guide. For multiple simultaneous requests inside one container, use @modal.concurrent and set its limits with resource constraints in mind, following Modal’s input-concurrency documentation.

Choose hardware before scaling llm hosting jobs

Use Modal’s GPU acceleration guide to make the hardware configuration explicit:

A GPU requires a valid payment method on file.
The documented gpu values are T4, L4, A10, L40S, A100, A100-40GB, A100-80GB, RTX-PRO-6000, H100/H100!, H200, B200/B200+ and B300.
For inference rather than training, Modal recommends starting with L40S, which has 48 GB of GPU memory.
To request multiple GPUs in one container, append :n to the GPU value. B300, B200, H200, H100, A100, L4, T4 and L40S support up to 8 GPUs per container, while A10 supports up to 4.
A request for more than 2 GPUs per container usually has a longer wait, and all GPUs assigned to that container are attached to the same physical machine.
B300 requires CUDA 13.1 or later, so the container image and libraries must be compatible with CUDA 13 before requesting it.

Modal also supports an ordered fallback list. The documented example is:

@app.function(gpu=["H100", "A100-40GB:2"])

Modal tries the preferred GPU type first and then attempts the less-preferred alternatives. Treat this as allocation flexibility, not as a way to create more parallel jobs: Function.map() handles job-level concurrency, while multi-GPU configuration changes the resources available inside each container.

Submit independent LLM jobs and preserve result order

Use Function.map() when each input can be processed independently. Keep a stable request ID beside every input, then pair each returned value with the ID at the same iterable position. This follows the ordered-result behavior documented in Modal’s scaling guidance.

By default, an exception from any mapped call propagates to the caller. If your application must continue collecting partial results, set return_exceptions=True and represent each failure as a returned result. Decide that policy before processing a large collection; otherwise, one failed call can interrupt result handling.

The autoscaler adds containers when it has no capacity for new inputs and removes them when resources are idle. Its main controls are:

max_containers: the upper limit for the Function.
min_containers: the number of containers kept warm while the Function is inactive.
buffer_containers: spare capacity maintained while the Function is active so additional inputs need not wait for a new container.
scaledown_window: the maximum idle duration, in seconds, before an individual container is scaled down.

A single Function has a documented hard limit of 4,000 concurrent containers. Workspace-wide container and GPU concurrency limits depend on the plan, so max_containers is not the only capacity boundary to check.

For a deployed Function, the documented autoscaler update form is:

f.update_autoscaler(min_containers=2, max_containers=10)

The literal values are an example rather than a capacity recommendation. An update made this way is temporary: the autoscaler settings return to the Function decorator’s values at the next deployment.

Run multiple requests inside each GPU container

Job fan-out and input concurrency solve different problems. Use Function.map() to process independent inputs across the Function’s container pool. Use @modal.concurrent when each active container should handle more than one input simultaneously.

By default, a container handles one input at a time. The documented decorator form is:

@modal.concurrent(max_inputs=100)

Treat 100 as a syntax example, not as a universally safe LLM setting. max_inputs limits simultaneous inputs within one container, while target_inputs is the concurrency level the autoscaler aims for. Existing containers may temporarily burst above the target up to max_inputs.

When demand exceeds max_inputs, Modal can add containers, but additional inputs may queue while those containers cold start. Modal recommends choosing target_inputs from your desired latency and choosing max_inputs from resource constraints, including avoiding GPU out-of-memory failures.

The execution model also matters. Modal runs concurrent inputs to synchronous Functions on separate threads, so the implementation must be thread-safe. Async Functions use asyncio tasks on one thread instead. Modal identifies continuous batching on GPU containers, including with vLLM, as a suitable use case for input concurrency.

Group requests only when dynamic batching fits

Dynamic batching is different from submitting independent mapped inputs. It waits for the lesser of a configured time or request count before executing a batch. The documented decorator is:

@modal.batched(max_batch_size=2, wait_ms=1000)

According to Modal’s dynamic batching guide, max_batch_size controls the request-count threshold and wait_ms controls the time threshold. Use this mechanism when combining requests is part of the workload design. It should not be treated as a synonym for Function.map() or @modal.concurrent, because it deliberately changes the grouping boundary before execution.

The values above demonstrate the documented syntax; they are not a throughput or latency promise. Select the boundaries according to the batching behavior your application requires.

Reduce cold starts without hiding queueing

A cold start occurs when no ready container exists and Modal must start a new one. Modal associates this condition with higher latency, and its cold-start guidance identifies both waiting for a warm container and first-invocation initialization as contributing factors.

Modal says containers boot in about one second, but global-scope work such as imports and modal.enter methods must finish before the container is considered warm. No inputs are routed to it until all entry methods complete. The first recommended fix for slow initialization is to download model weights ahead of time rather than during container startup.

Moving work into an entry method does not remove the latency; it shifts that work into the warm-up period. Where feasible, Modal recommends reading large files concurrently rather than sequentially. This separates initialization work from steady-state inference instead of hiding it inside the first request.

The same guidance sets the default maximum idle time at 60 seconds and allows scaledown_window to be configured from 2 seconds to 20 minutes. Increasing the window can reduce the chance of a subsequent cold start, but resources reserved by an idle container are still billed. Choose the window as a readiness-versus-idle-use tradeoff rather than assuming additional warm capacity is free.

Check the default routing region

Before sending production payloads, review their route. As documented in Modal’s region selection guide, Function inputs route through Modal’s servers in Virginia, USA (us-east) by default before being sent to a container. If prompts, documents or other payloads have routing requirements, check that default against the current region documentation before deployment.

Verify results and find where the workflow goes wrong

Validate the workflow with representative non-production inputs. Preserve each request ID, confirm that the number of returned values matches the submitted inputs, and compare IDs by result position. Exercise both the default exception path and return_exceptions=True on a controlled input set so the caller’s failure behavior is known before concurrent processing begins.

Use these checks while tuning:

Decide whether a change belongs to job-level fan-out or per-container input concurrency before increasing both together.
Derive concurrency values from your model’s resource use and desired latency instead of copying documentation examples as production settings.
Record queue waiting, first-invocation initialization and steady-state work separately so a cold start is not mistaken for slow inference.
Treat partial failures as a designed result type, or ensure exception propagation is acceptable to the calling workflow.
Review idle-container use whenever you lengthen the scale-down window.
Validate the default input route against your production payload requirements.

These steps are compiled from the official documentation linked in the text. Nothing here was run or measured, so check the current documentation and version before use.