AI DeploySelf-hosted LLMGet it deployed

Model Long-Term vs On-Demand GPU Costs

Compare long-term GPU commitments with on-demand charges using a workload cost model that includes storage, networking, interruptions, and constraints.

For llm hosting, model the long-term option as commitment or prepaid charges plus usage that does not receive the applicable discount, and model the On-Demand option as instance runtime plus attached services. AWS describes Savings Plans as commitments to consistent usage measured in USD per hour for a 1- or 3-year term, while EC2 On-Demand Instances are billed by the second, as explained in Amazon EC2 billing and purchasing options. Compare both options over the same workload, Region, storage footprint, and network path; a lower GPU figure alone does not establish a lower total.

Prepare a comparable workload and quote

Create a worksheet row for each model, GPU, and Region combination. Record the model revision and precision, typical input and output lengths, concurrency, latency target, operating schedule, GPU SKU, Availability Zone, storage, and traffic path. Draw workload assumptions from your own records and rates from the current provider quote or bill; otherwise, the two totals may describe different work.

Keep the GPU SKU and card variant exact. NVIDIA’s H100 specification table lists 80 GB memory for H100 SXM and 94 GB for H100 NVL; maximum TDP is up to 700W configurable for SXM and 350–400W configurable for PCIe NVL. Record those distinctions rather than merging variants, but do not turn memory or TDP directly into a cloud price.

Make the load comparable as well. The MLPerf Inference Rules require every submission to use LoadGen, which generates queries by scenario, tracks latency, validates accuracy, and computes final metrics. The Server scenario uses a Poisson distribution and the maximum Poisson throughput supported; Offline sends all samples at the start in a single query and reports throughput.

Build an llm hosting cost model

Use the uppercase labels below as worksheet fields, not API keys or provider commands:

WORKLOAD_ID = YOUR_VALUE
GPU_SKU = YOUR_VALUE
REGION = YOUR_VALUE
AVAILABILITY_ZONE = YOUR_VALUE
PURCHASE_OPTION = YOUR_VALUE
TERM = YOUR_VALUE
PLANNING_HORIZON = YOUR_VALUE

ON_DEMAND_INSTANCE_COST = SUM(ACTUAL_INSTANCE_CHARGE_LINES)
LONG_TERM_BASE_COST = SUM(COMMITMENT_OR_PREPAID_CHARGE_LINES)
NON_MATCHED_INSTANCE_COST = SUM(NON_MATCHED_INSTANCE_CHARGE_LINES)

TOTAL_ON_DEMAND =
  ON_DEMAND_INSTANCE_COST
  + STORAGE_COST
  + PUBLIC_IPV4_COST
  + NAT_GATEWAY_COST
  + DATA_TRANSFER_COST
  + OTHER_LINKED_COST

TOTAL_LONG_TERM =
  LONG_TERM_BASE_COST
  + NON_MATCHED_INSTANCE_COST
  + STORAGE_COST
  + PUBLIC_IPV4_COST
  + NAT_GATEWAY_COST
  + DATA_TRANSFER_COST
  + OTHER_LINKED_COST

NET_CHANGE = TOTAL_LONG_TERM - TOTAL_ON_DEMAND

On the provider’s purchase page, read the exact GPU, Region or Availability Zone, term, commitment unit, cancellation terms, and discount scope. From the quote or invoice, collect:

  • Instance charges for covered, committed, prepaid, and non-matched usage.
  • Storage, public IPv4, NAT gateway, and data-transfer charges.
  • Any other linked resources that remain billable during the comparison period.

Use only the rows that apply to the selected purchase model. A positive NET_CHANGE means the long-term model is more expensive over the selected horizon; a negative value means it is lower. When billing has minimum durations or fine-grained metering, use the invoice’s charge lines rather than inventing a rounding rule.

For a purchase option whose quote contains an up-front charge, calculate break-even only against a positive expected monthly saving:

UP_FRONT_COST = YOUR_VALUE
EXPECTED_MONTHLY_SAVING = YOUR_VALUE
BREAK_EVEN_MONTHS = UP_FRONT_COST / EXPECTED_MONTHLY_SAVING

Keep storage and networking unchanged on both sides of this calculation. Do not apply the formula unless the quote actually contains an up-front amount.

Define what the long-term option buys

The Compute Savings Plans overview states that Compute Savings Plans apply to EC2 usage regardless of instance family, size, OS, tenancy, or Region. Keep the exact launched GPU in the worksheet even though the commitment has broader coverage.

AWS’s Reserved Instances overview describes a purchase commitment to a specific instance type and Region for a 1- or 3-year term. It is a billing discount applied to matching On-Demand usage rather than a physical instance; the purchase cannot be cancelled, does not renew automatically, and leaves the instance running at On-Demand rates after expiry.

An EC2 Capacity Reservations guide says a reservation covers a specific Availability Zone for any duration and can be cancelled at any time to stop charges. Its pricing and billing documentation says reservations are charged at the equivalent On-Demand rate whether instances run or not, use per-second billing with a 60-second minimum, and can receive Savings Plans and Regional Reserved Instance discounts.

The Capacity Blocks for ML documentation says Capacity Blocks reserve clusters of GPU instances, cannot be cancelled, and end at 11:30 AM UTC. The related pricing and billing documentation says the block is charged up front when reserved and does not receive Savings Plans or Reserved Instance discounts.

Google’s Compute Engine committed use discounts overview describes discounted VM prices in return for a 1-year or 3-year commitment. These choices commit different things—usage, capacity, or cluster availability—so compare the actual obligation rather than treating every long-term purchase as interchangeable.

Add charges that remain after GPU shutdown

AWS says EC2 On-Demand billing counts only seconds in the running state and applies a 60-second minimum, according to the On-Demand Instances documentation. The EC2 instance-state documentation adds that stopped instances are not billed for instance usage, each stop-to-running transition begins a new billing period with a one-minute minimum, and resources such as EBS volumes and Elastic IP addresses can incur charges regardless of instance state.

Address charges also need explicit lines. AWS’s VPC IP addressing documentation states that every public IPv4 address is charged, including addresses associated with running instances and Elastic IPs. The NAT gateway pricing page documents charges for each hour the gateway is available and each gigabyte processed; it also identifies same-Availability-Zone placement as reducing cross-zone transfer charges.

In the AWS data-transfer charge documentation, cross-Availability-Zone traffic inside a Region appears under Region-DataTransfer-Regional-Bytes, while transfer from AWS to the internet appears under Region-DataTransfer-Out-Bytes. Inbound internet transfer to AWS is not charged, while inter-Region charging applies to data leaving the source Region. Use those categories to connect the worksheet’s transfer line to the invoice.

Keep Spot outside the two baselines

Do not silently treat Spot capacity as On-Demand capacity or include it in a Savings Plans saving. The EC2 Spot Instances documentation says Spot Instances are not covered by Savings Plans; when On-Demand capacity is unavailable, an On-Demand launch receives an insufficient capacity error, while a Spot request keeps retrying.

Spot also needs an interruption allowance. The Spot interruption notice documentation says the notice is issued two minutes before an instance is stopped or terminated, is delivered on a best-effort basis, and should be checked every 5 seconds. Through IMDSv2, the documented path is:

http://169.254.169.254/latest/meta-data/spot/instance-action

While an interrupted Spot Instance is stopped, its preserved EBS volumes remain billable, as explained in the interrupted Spot billing documentation. Add replacement compute and repeated supporting charges to the worksheet rather than treating the lower initial rate as the whole cost.

Verify before committing

Before accepting a long-term purchase, complete the reconciliation:

  • Match the GPU SKU, Region, Availability Zone, term, and operating schedule across the quote and invoice.
  • Map every formula field to an actual charge line and separate up-front, recurring, and usage-based amounts.
  • Recalculate the totals with low and high operating schedules taken from your own workload records.
  • Confirm the current start, end, cancellation, and discount conditions on the provider’s purchase page.
  • Keep Spot interruption and retry costs outside both the committed and On-Demand baselines.

Check launch constraints as well. The On-Demand Instances documentation says quotas are counted in vCPUs rather than instances, lists the default quota for G and VT GPU families as 0 vCPUs with adjustment available, and states that unused Capacity Reservations still count toward the quota.

These steps are compiled from the official documentation linked in the text and should be checked against the current version before use. No price quote or benchmark result is supplied here.

Sources