AI DeploySelf-hosted LLMGet it deployed

Audit GPU Cloud Waste for Self-Hosted LLMs

Learn how to audit AWS costs for self-hosted LLMs, including storage that may remain charged after compute stops, then remove waste and add controls.

Audit GPU cloud waste by separating compute, storage, snapshots, public IP, data transfer, and serving configuration instead of treating accelerator usage as the whole bill. For llm hosting, attach every charge to a resource and workload, classify it as serving, retained, scheduled, or unowned, and remove only what has no approved purpose. This walkthrough uses AWS as the documented worked example; on another provider, verify its current billing and state rules before changing resources.

How should you reconcile an llm hosting bill?

Start with the provider invoice or cost export and a resource inventory. Record:

  • Account, billing period, provider, and Region.
  • Resource ID, current state, and attached services.
  • Workload, technical owner, and business owner.
  • Compute, storage, snapshot, public IP, and data-transfer lines.
  • Start, stop, interruption, retention, and rollback events.
  • The reason each retained resource still has value.

Read the provider’s own pages for state-based billing, storage and snapshot deletion, public-address charges, transfer classifications, and alert update timing. Fill every placeholder below from your bill or approved quote rather than estimating it:

TOTAL_CLOUD_COST = COMPUTE_COST + STORAGE_COST + SNAPSHOT_COST + PUBLIC_IP_COST + TRANSFER_COST + ATTACHED_SERVICE_COST

WASTE_GAP = TOTAL_CLOUD_COST - APPROVED_RUN_COST - APPROVED_IDLE_BUFFER

A positive WASTE_GAP is a review queue, not automatically recoverable spend. First resolve missing lines, shared infrastructure, and resources required for rollback.

How do you separate compute state from dependent charges?

Under Amazon EC2 instance state changes, a stopped EC2 instance is not billed for instance usage, while EBS volumes and Elastic IP addresses can still incur charges; each stopped-to-running transition starts a new billing period with a one-minute minimum, and an instance being hibernated is billed while in the stopping state.

Classify each GPU instance as actively serving, scheduled, retained, or unowned. For scheduled workloads, record the expected restart and review every attached resource separately. For troubleshooting restarts, record each stop-to-running transition so repeated tests are visible in the audit.

Do not mark an instance “deleted” merely because it is stopped. Mark the compute state and the disposition of its storage, addresses, snapshots, and networking as separate fields.

What storage dependencies can survive cleanup?

AWS Billing’s guidance on unexpected charges says stopping an instance can still leave storage charges, deleting a volume does not delete its associated snapshots, and a load balancer must be terminated before its registered EC2 instances are deleted.

Build the reverse dependency chain before approving removal:

  • GPU instance to attached volumes.
  • Volume to retained snapshots.
  • Load balancer to registered instances.
  • Instance to its public addresses.
  • Deployment to rollback configuration and retained data.

Cleanup commands are resource-specific, so use the provider’s current console or API instructions rather than applying a generic delete sequence. Record why every retained object remains and which future bill line it is expected to affect.

Where can network cost hide?

Use AWS’s data-transfer report guide to classify charges: cross-Availability-Zone transfer inside a Region appears under DataTransfer-Regional-Bytes, AWS-to-internet transfer appears under DataTransfer-Out-Bytes, internet-to-AWS inbound transfer is not charged, and inter-Region charges attach to data leaving the source Region rather than entering the destination Region.

Keep those categories separate in the worksheet:

TRANSFER_EXPOSURE = CROSS_AZ_COST + INTERNET_EGRESS_COST + INTER_REGION_EGRESS_COST

Do not net an uncharged inbound direction against a charged outbound direction. Attach each transfer row to the workload that caused it and decide whether the pattern is expected.

Give every Elastic IP its own inventory row regardless of the linked compute state. An idle or rollback-only address can otherwise disappear from a review that starts with GPU instances.

How should commitments and alerts be reviewed?

Before renewing GPU capacity, read the current quote and contract for the covered service, commitment term, billing unit, included fees, and change or cancellation conditions. Fill the worksheet only from those documents:

NET_COMMITMENT_COST = QUOTED_COMMITMENT + QUOTED_FEES - CONFIRMED_CREDIT

This is an audit formula, not a provider contract formula. Compare the commitment with observed schedules and approved rollback capacity; do not infer remaining value from GPU activity alone.

For AWS, AWS Budgets guidance documents alerts for both actual and forecast spend, but Budgets data is updated only up to three times a day, typically 8–12 hours after the previous update. Set a threshold from your own approved cost worksheet and name the person responsible for responding instead of treating the alert as an immediate shutdown control.

Is the serving configuration using the GPU efficiently?

The vLLM vllm serve CLI reference defines --gpu-memory-utilization as the fraction of GPU memory used by the model executor, documents 0.92 as the default, and states that the limit applies per vLLM instance.

Record the effective argument and whether it was explicit:

--gpu-memory-utilization 0.92

This is an argument fragment, not a standalone command. The documented default is not a capacity recommendation for your model. Map each serving process to its GPU assignment, model, explicit or implicit setting, and owner. Because the limit is instance-specific, one service’s value does not describe every process sharing that GPU. Change it only after workload-specific validation, and preserve the previous serving configuration for rollback.

How can monitoring support the audit?

The Prometheus getting-started guide uses a 15s example global.scrape_interval, lists targets through scrape_configs, stores its database in ./data by default through --storage.tsdb.path, and starts Prometheus with the following command:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'YOUR_VALUE'
    static_configs:
      - targets: ['YOUR_VALUE']

./prometheus --config.file=prometheus.yml

Replace placeholders only with a target you operate and have verified exposes the metrics you expect. The scrape configuration alone does not prove that a GPU is idle or that a cloud charge should stop. Use it to correlate workload activity with the bill, and include the Prometheus data path in the storage-cleanup inventory so monitoring does not become unowned residue.

How do you verify cleanup and avoid repeat waste?

Repeat the same worksheet after approved changes:

  • Compare resource IDs and states with the previous inventory.
  • Recalculate the cost and waste-gap formulas from new billing data.
  • Confirm that every retained dependency has an owner and reason.
  • Verify that the Prometheus target produces current samples.
  • Confirm that each budget alert has a threshold, recipient, and response action.

Where current documentation does not state when a charge ends, use the next available billing record rather than assuming the change was immediate.

Common repeat failures are treating stopped compute as deleted, reviewing only the GPU line, retaining snapshots after volume cleanup, omitting public addresses from the inventory, offsetting transfer categories, reading one vLLM process setting as the whole GPU’s state, and leaving monitoring data without an owner.

These steps are compiled from the official documentation linked in the text, and nothing was run or measured for this article; check the current versions before use.

Sources