AI DeploySelf-hosted LLMGet it deployed

Modal vs. AWS Lambda GPU for LLM Inference

Compare Modal's documented GPU attachment with AWS Lambda runtime limits, then choose a practical Python LLM inference setup with a transparent cost checklist.

For AI deployment, Modal’s GPU acceleration guide documents attaching a GPU through a Function’s gpu argument; its documented values are T4, L4, A10, L40S, A100, A100-40GB, A100-80GB, RTX-PRO-6000, H100/H100!, H200, B200/B200+ and B300, and it requires a valid payment method. The same guide recommends L40S for inference with 48 GB of GPU RAM and accepts ordered fallback lists plus :n GPU counts, while AWS Lambda’s quotas document a 900-second timeout, memory from 128 MB to 10,240 MB in 1-MB increments with proportional CPU, deployment-package and container-image ceilings, but do not establish GPU assignment. The practical answer is asymmetric: Modal has a documented GPU configuration path, while Lambda remains unverified as a GPU-backed LLM endpoint until AWS publishes the missing allocation, memory and billing details.

The AI deployment decision the documentation supports

Use capability—not provider preference—as the gate. Modal has enough documented information to begin endpoint design; Lambda does not have documented GPU allocation in the linked quota material.

Before choosing, require a confirmed GPU model and memory allowance, an official serving-project launch method and endpoint path, a fit check for model weights and runtime, and a cost method covering loading, idle capacity and requests. An unresolved GPU assignment or memory answer blocks Lambda deployment for this workload.

Prepare the Python inference service

Choose a Python serving project before configuring provider resources. From that project’s current official documentation, record the exact model-loading command, server flags, endpoint path, client command and supported runtime versions. The provider pages cited here do not define a generic LLM endpoint, so this article does not invent one.

Prepare the following inputs:

  • The exact model artifact, runtime dependencies and input/output contract.
  • The model-weight and working-memory requirements you need to compare with the selected GPU memory.
  • The maximum request duration and response size expected from the endpoint.
  • The current provider rates and billing categories you will use in the cost formula.
  • A cold-start policy based on the traffic pattern you actually expect.

The Modal configuration forms below are alternatives, not decorators to stack:

@app.function(gpu="L40S")

@app.function(gpu=["H100", "A100-40GB:2"])

@app.function(gpu="H100:8")

Use a GPU count only when the model and runtime genuinely require that configuration. Keep fallback order explicit so the accepted hardware remains visible in code.

Configure Modal’s scaling controls

According to Modal’s Scaling out guide, every Function has its own autoscaling container pool and scales to zero by default when no inputs remain. The autoscaler adds containers when it lacks capacity for new inputs and removes containers when resources are idle.

The documented settings have separate purposes:

  • min_containers sets the number kept warm while the Function is inactive.
  • max_containers sets the Function’s container ceiling.
  • buffer_containers maintains spare capacity while the Function is active so additional inputs need not queue for another container.
  • scaledown_window sets the maximum idle duration, in seconds, before a container is scaled down.

A deployed Function can be updated without redeploying:

f.update_autoscaler(min_containers=2, max_containers=10)

An update made through this call does not permanently replace the decorator configuration: the autoscaler settings return to their decorator values at the next App deployment. Keep that behavior in mind when comparing a live capacity change with the configuration that will be reproduced later.

Compare cold-start policies before choosing a route

Modal’s cold-start documentation defines a cold start as the need to start a container when none is ready; it associates the delay with waiting for a warm container and initialization during the first invocation. Modal describes container boot as about one second, but global-scope code, imports and modal.enter methods must finish before the container is warm, and it recommends downloading model weights ahead of time rather than during boot.

AWS’s provisioned concurrency documentation defines provisioned concurrency as pre-initialized execution environments intended to reduce cold starts. It says those environments are ready to respond immediately, while the option runs continually and incurs charges separate from initialization and invocation; initialization is charged even when the environment never serves a request.

These are different policies. Modal’s warm-container controls and AWS provisioned concurrency both seek readiness, but their billing and initialization paths must be evaluated separately. Neither provider’s startup description should be treated as an end-to-end LLM latency promise.

Calculate provider charges from your own quote

Modal’s pricing documentation meters CPU, memory and GPU compute by the second. Billable GPU time includes application loading, input processing and the default idle keep-alive window of 60 seconds after the last input; CPU and memory are charged on the higher of requested or actual use, and no minimum usage time per instance is required. Once the application has scaled down to zero containers, Modal says no further compute charge applies.

Read the provider’s current pricing page and your account details for:

  • The current GPU, CPU and memory compute rates.
  • The monthly compute allowance available to the Workspace.
  • GPU time spent loading the application.
  • GPU time spent processing inputs.
  • GPU time spent in the keep-alive window.

Replace every placeholder from those inputs:

MODAL_ESTIMATED_CHARGE =
  GPU_RATE_FROM_CURRENT_QUOTE × BILLED_GPU_SECONDS
  + CPU_BILLABLE_AMOUNT_FROM_CURRENT_QUOTE
  + MEMORY_BILLABLE_AMOUNT_FROM_CURRENT_QUOTE
  - MONTHLY_COMPUTE_ALLOWANCE_APPLIED_FROM_CURRENT_ACCOUNT

BILLED_GPU_SECONDS =
  APPLICATION_LOAD_GPU_SECONDS
  + INPUT_PROCESSING_GPU_SECONDS
  + KEEP_ALIVE_GPU_SECONDS

CPU_BILLABLE_AMOUNT = HIGHER_OF_REQUESTED_AMOUNT_OR_ACTUAL_AMOUNT
MEMORY_BILLABLE_AMOUNT = HIGHER_OF_REQUESTED_AMOUNT_OR_ACTUAL_AMOUNT

For AWS Lambda, take initialization, invocation and provisioned concurrency amounts from your own quote or bill:

AWS_LAMBDA_ESTIMATED_CHARGE =
  INITIALIZATION_CHARGES_FROM_QUOTE_OR_BILL
  + INVOCATION_CHARGES_FROM_QUOTE_OR_BILL
  + PROVISIONED_CONCURRENCY_CHARGES_FROM_QUOTE_OR_BILL

Leave GPU allocation charges out of that formula until an official AWS document and quote define them. Do not compare a Modal GPU estimate with an AWS Lambda CPU-function estimate as though both represent the same workload.

Verify the deployment and catch common errors

Use this order before declaring the endpoint ready:

  • GPU evidence: Confirm the exact provider allocation. For Lambda, require current official documentation that maps the Function to a GPU; ordinary CPU, memory and timeout quotas are not substitutes.
  • Model fit: Compare weights, activations and runtime memory with the actual GPU allocation. A provider’s starting recommendation is not proof that your model fits.
  • Endpoint evidence: Use the selected serving project’s documented route and client command. Do not invent a path or assume that successful Function configuration creates an HTTP endpoint.
  • Behavior: Send a fixed test input through that documented interface, then exercise the concurrency pattern you intend to use. Record queueing, errors and responses in your own environment before making capacity or performance claims.
  • Cost: Reconcile the resulting invoice with the formula. Check application loading, Modal keep-alive time and any continuously billed AWS provisioned concurrency rather than comparing invocation totals alone.

These steps are compiled from the official documentation linked in the text; nothing was run or measured, and the steps should be checked against the current version before use.

Sources