Calculate Total Cost of Ownership for Self-Hosted LLMs

Build a TCO worksheet that separates GPU hardware, bandwidth, cloud extras, operations, maintenance, staffing, and energy.

Calculate the total cost of ownership for a self hosted llm by adding one-time capital costs, period operating costs, and exit costs for each option over the same workload period. Keep owned hardware, rented GPU time, storage, bandwidth, power, cooling, maintenance, monitoring, CI/CD, and staff time in separate worksheet rows, filling every rate from your own quote, bill, or meter. A lower purchase price does not prove a lower TCO when uptime, utilization, data transfer, staffing, or exit assumptions differ.

Define the self hosted llm TCO boundary

Set the analysis period, expected workload, availability target, deployment location, and accuracy required from the estimate. Record running and idle time, model and dataset storage, checkpoint frequency, network traffic, recovery requirements, and the staff coverage needed to operate the service.

Keep cash TCO and economic TCO separate. The cash view records payments made during the period. The economic view allocates qualifying capital costs over their useful life. Do not add the full acquisition cost and its depreciation allocation to the same total.

Build a source-backed input sheet

Create a worksheet row for each charge or cost category and capture:

Quantity, unit, rate, currency, and billing condition.
Fixed or variable classification.
Start and end dates, commitment term, and allocation method.
Official page, quote, invoice, or meter used as evidence.
Date checked and the person responsible for renewal.

A blank cell is not a zero-cost item. Enter zero only when the current documentation or quote explicitly supports it; otherwise request written confirmation.

Record hardware, power, and physical fit

As a worksheet example rather than a purchasing recommendation, NVIDIA’s GeForce RTX 4090 specifications list 24 GB of GDDR6X memory, 450 W total graphics power, and 850 W required system power for the Founders Edition. The same specification lists a 3-slot card measuring 304 mm by 137 mm. This illustrates why memory, board power, whole-system power, and physical fit belong in separate worksheet fields.

Collect acquisition quotes for the GPU, chassis, CPU, memory, storage, networking, mounting hardware, uninterruptible power equipment, and cooling equipment. Treat published power ratings as design inputs, not measured electricity consumption.

Energy needs a separate facility calculation. Google’s Power usage effectiveness explainer defines PUE as the relationship between non-computing overhead energy and the energy powering IT equipment. It gives the example that a PUE of 2.0 means one additional watt of overhead for each watt of IT power, while a value closer to 1.0 means nearly all energy supports computing.

YOUR_FACILITY_KWH = YOUR_IT_EQUIPMENT_KWH × YOUR_PUE
YOUR_ENERGY_COST = YOUR_FACILITY_KWH × YOUR_ELECTRICITY_RATE

Use a suitable facility boundary and an attributable meter or bill. Do not insert a GPU power rating as though it were metered consumption, and do not charge cooling energy again if it is already included through PUE.

Record cloud compute, storage, and transfer

For AWS, the EC2 instance-state documentation states that a stopped instance is not billed for instance usage, while EBS volumes and Elastic IP addresses are charged regardless of instance state. Each stop-to-running transition starts a new instance billing period with a one-minute minimum.

AWS NAT gateway pricing documentation states that a NAT gateway is charged for each hour it is available and for each gigabyte it processes. It also recommends placing resources that exchange significant traffic in the same Availability Zone as the gateway to reduce data-transfer charges.

YOUR_NETWORK_COST =
  (YOUR_TRANSFER_GB × YOUR_TRANSFER_RATE)
  + (YOUR_NAT_HOURS × YOUR_NAT_HOUR_RATE)
  + (YOUR_NAT_GB × YOUR_NAT_DATA_RATE)

Read the provider’s current billing page and invoice for compute, GPU, storage, backups, addresses, load balancers, network transfer, monitoring, logs, support, and shutdown-related charges. Keep each item visible even when the provider combines several charges under one invoice line.

Record operations and people

Current provider and project documentation may not settle every operations cost. Check the applicable pricing and billing pages for monitoring, logging, CI/CD runners, security controls, backups, model deployment, upgrades, and support. If inclusion is unclear, request a quote rather than assuming the service is free.

Add staff time for administration, model and dataset operations, incident response, capacity planning, security, and supplier management.

YOUR_STAFF_COST =
  (YOUR_STAFF_HOURS × YOUR_STAFF_RATE)
  + YOUR_TRAINING_COST
  + YOUR_OPERATIONS_TOOLS_COST

Calculate totals without double counting

Apply only the rows relevant to the deployment being evaluated.

YOUR_ONE_TIME_COST =
  YOUR_ACQUISITION_COST
  + YOUR_INSTALLATION_COST
  + YOUR_FACILITY_CAPACITY_COST
  + YOUR_INITIAL_BACKUP_COST

YOUR_PERIOD_OPERATING_COST =
  YOUR_DEPRECIATION_ALLOCATION
  + YOUR_ENERGY_COST
  + YOUR_NON_ENERGY_FACILITY_COST
  + YOUR_NETWORK_COST
  + YOUR_STORAGE_COST
  + YOUR_BACKUP_COST
  + YOUR_MONITORING_COST
  + YOUR_LOGGING_COST
  + YOUR_CICD_COST
  + YOUR_MAINTENANCE_COST
  + YOUR_STAFF_COST
  + YOUR_INCIDENT_RESPONSE_COST

YOUR_CASH_TCO =
  YOUR_ONE_TIME_COST
  + YOUR_PERIOD_OPERATING_COST
  + YOUR_EXIT_COST

For an economic-cost view, use an approved depreciation schedule rather than also charging the full acquisition price.

YOUR_DEPRECIATION_ALLOCATION =
  (YOUR_DEPRECIABLE_BASE - YOUR_RESIDUAL_VALUE)
  ÷ YOUR_LIFE_YEARS

Residual value, useful life, and tax treatment depend on the relevant accounting policy and jurisdiction; have the responsible accountant confirm them. Exit costs can include decommissioning, data transfer, restoration work, contract commitments, and migration.

Compare purchase commitments and exit exposure

The AWS EC2 purchasing documentation says On-Demand Instances are billed by the second. It describes Savings Plans as commitments to consistent usage measured in USD per hour for a 1- or 3-year term, Reserved Instances as commitments to a specific instance configuration and Region for a 1- or 3-year term, and Capacity Blocks as an option for reserving a cluster of GPU instances.

For any commitment, read the quote for the eligible usage, configuration, billing unit, start and end dates, unused commitment, termination conditions, and any refund treatment. Compare the commitment with the same workload and analysis period purchased through an available alternative.

YOUR_COMMITMENT_NET =
  YOUR_CONTRACT_COST
  - YOUR_COUNTERFACTUAL_COST

Do not label the difference as a saving until the counterfactual uses the same workload, utilization, region, and service period. Include exit exposure rather than assuming unused capacity can be recovered without charge.

Track energy and carbon as separate outputs

The GHG Protocol Scope 2 Guidance defines Scope 2 as indirect emissions from the generation of purchased or acquired electricity, steam, heat, or cooling. Its location-based method uses average emissions intensity for the grid where consumption occurs, while its market-based method uses factors associated with contractual instruments. Where contractual instruments apply, it requires both methods to be reported and labelled.

YOUR_LOCATION_BASED_EMISSIONS =
  YOUR_ELECTRICITY_KWH × YOUR_GRID_AVERAGE_FACTOR

YOUR_MARKET_BASED_EMISSIONS =
  YOUR_ELECTRICITY_KWH × YOUR_CONTRACTUAL_FACTOR

Keep these results separate rather than adding them together. Carbon becomes a monetary TCO line only if your organization has adopted an approved internal carbon price; if so, show that assumption and the resulting amount as a separate cost category.

Verify the worksheet before approval

Confirm that every option uses the same workload, availability target, analysis period, and allocation rules.
Trace each rate to a current official page, quote, invoice, or attributable energy record.
Reconcile cloud totals against the bill categories, including storage and transfer outside compute charges.
Check that PUE-based energy and separately billed facility services do not overlap.
Verify that bundled services have documentary support and that unconfirmed items have not been entered as zero.
Separate cash payments, depreciation, taxes, and internal carbon assumptions into distinct views.
Reconcile low, expected, and high scenarios against the same formulas; change assumptions rather than rebuilding each scenario.

Common failure points

The most frequent errors are comparing GPU purchase prices alone, treating nameplate power as measured consumption, omitting idle facility overhead, and ignoring staff time. Others include treating documentation silence as free service, assuming a stopped compute resource makes the environment cost nothing, mixing committed and variable prices, double-counting depreciation, and comparing carbon results produced with different accounting methods.

These steps are compiled from the official documentation linked in the text; nothing was run or measured for this article, so check the current documentation and your own bill or quote before use.