Inspect Used GPUs for a Self-Hosted LLM Deployment
Inspect a used GPU with nvidia-smi -q, which reports all attributes, then verify documentation, compliance, memory, power and container access.
A used GPU can support a self hosted llm deployment only after you verify board identity, published specifications, driver state, licensing fit, and container visibility. Start with nvidia-smi -q, compare the reported serial with the board label, inspect power and physical fit, and then verify the card from a GPU-enabled container. Treat this as an acceptance workflow, not proof of workload performance; nothing was run or measured for this article.
Questions a self hosted llm buyer should answer
Create an acceptance record for every board before powering it. Record the exact model, board serial, seller identity, provenance and service history as presented, included parts, and actual return or warranty terms. Photograph the labels and connectors.
Record the target model and runtime’s documented memory requirement. Check the chassis slot, card clearance, cooling path, power capacity, and any required data or auxiliary cables. Treat missing seller evidence as unknown rather than as a pass.
Match the exact board variant
Published specifications are comparison points, not proof that a used board is healthy or matches its listing. Check the official page for each exact variant:
| GPU | Published memory | Published power | Published mechanical details |
|---|---|---|---|
| GeForce RTX 4090 | 24 GB GDDR6X | 450 W total graphics power; 850 W required system power | 3 slots; 304 × 137 mm |
| GeForce RTX 5090 | 32 GB GDDR7 | 575 W total graphics power; 1000 W required system power | 2 slots; 304 mm long |
| NVIDIA L4 | 24 GB | 72 W maximum TDP | 1-slot low-profile PCIe card |
| NVIDIA L40S | 48 GB GDDR6 with ECC | 350 W maximum consumption | Dual-slot, passively cooled; 16-pin power connector |
| NVIDIA A100 | 80 GB HBM2e | 300 W maximum TDP for PCIe; 400 W for SXM | PCIe and SXM variants |
| NVIDIA H100 | 80 GB for SXM; 94 GB for NVL | Up to 700 W configurable for SXM; 350–400 W configurable for PCIe NVL | SXM or dual-slot air-cooled PCIe |
| NVIDIA H200 | 141 GB for SXM and NVL | Up to 700 W configurable for SXM; up to 600 W configurable for NVL | SXM and NVL variants |
Check the physical board label instead of relying on the listing title. Memory capacity is only one constraint: the target software’s current documentation must support the exact GPU and its memory demand. Treat dimensions, slot width, connectors, and required system power as acceptance constraints. These published specifications cannot establish whether a used card has physical damage, prior overheating, or an unresolved fault history.
Run the documented inspection commands
The NVIDIA System Management Interface documentation defines -q as a full GPU-attribute view, uses N/A for unsupported data, says the reported board serial matches the number printed on the board, and documents comma-separated query properties, ECC-retired pages, row-remapper status, temperature, power, and fan fields. It also warns that reported fan speed is the intended speed rather than a measured speed.
Run the documented commands and retain their output:
nvidia-smi -q
nvidia-smi --query-gpu=pci.bus_id,persistence_mode
nvidia-smi --help-query-gpu
Use the full output to compare the board identity and review Retired Pages, Row Remapper, fan speed, GPU Current Temp, and power readings. Leave unsupported N/A fields explicitly unresolved; do not convert them into a successful check. Compare the serial in the output with the physical board marking before accepting the card.
An Xid message is an NVIDIA driver error report and can indicate a hardware, NVIDIA software, or application problem. On Linux, the Xid troubleshooting guide directs administrators to the kernel log and the exact marker NVRM: Xid.
Memory-retirement evidence also depends on the GPU generation. On boards using the earlier scheme, the driver can automatically retire pages containing bad memory cells. Starting with Ampere, row remapping replaces that scheme, requires a GPU reset to take effect, and persists for the life of the GPU.
Verify the Docker GPU path
Follow Docker’s GPU access prerequisites: install the NVIDIA drivers, reboot the system, and install the NVIDIA Container Toolkit. The NVIDIA Container Toolkit installation guide requires the host driver first, recommends installing it through the distribution’s package manager, and documents the configuration and daemon-restart commands:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
The first command updates /etc/docker/daemon.json so Docker can use the NVIDIA Container Runtime; the second restarts the Docker daemon. NVIDIA’s sample workload procedure provides this container verification command:
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
For a CUDA application, the system needs a CUDA-enabled GPU and a driver compatible with the CUDA Toolkit used to build it, according to NVIDIA’s CUDA compatibility explanation. The minor-version compatibility table lists these minimum driver versions: CUDA 13.x: >= 580, CUDA 12.x: >= 525, and CUDA 11.x: >= 450.
Check the licence and compliance boundary
NVIDIA’s GeForce software licence states in clause 2.8 that GeForce or Titan driver software is licensed for use only on GeForce or Titan hardware products you own and is not licensed for datacenter deployment. Clause 2.7 prohibits providing commercial hosting services with the software unless expressly granted. Do not assume that buying a used GeForce card grants the software rights needed for a hosted business service; obtain a written answer for the intended deployment.
The AICPA description of SOC identifies SOC reports as CPA-issued assurance reports for assessing outsourced-service risks; it describes SOC as an attestation rather than an AICPA-issued certificate, and its SOC 2 guide names security, availability, processing integrity, confidentiality, and privacy as possible examination categories. Ask a seller or hosting provider for the actual report, issuing CPA, and stated scope. Do not treat it as proof of a used GPU’s physical condition.
The GPU specification pages do not establish export classification. Obtain a classification for the exact part, origin, destination, and ownership arrangement from the seller and the relevant authority or qualified export counsel, using current rules.
Under the official GDPR text, processing in the context of an EU establishment can trigger the Regulation regardless of where processing occurs; when a provider acts as a processor, Article 28 requires sufficient guarantees, specified contract terms, documented instructions, authorised subprocessors, and end-of-service return or deletion, while the Regulation also addresses risk-appropriate security, breach handling, and third-country transfers. This is general information, not legal advice.
Add purchase and energy calculations
Request an itemised purchase quote and include every applicable line rather than comparing only the advertised GPU price:
YOUR_VALUE = card price + shipping + taxes and duties + adapter or cabling + tested replacement parts + support or warranty
Keep host upgrades, storage, networking, and operator time separate unless the seller’s quote includes them. Record the return and warranty terms separately so a contingent cost is not mistaken for a fixed charge.
The GHG Protocol Scope 2 guidance defines Scope 2 as indirect emissions from generating purchased or acquired electricity, steam, heat, or cooling; its location-based method uses average grid emissions intensity, its market-based method uses factors from contractual instruments, and companies operating in markets with contractual instruments must label both results through dual reporting.
Fill the factors and consumption values from your own records:
location-based Scope 2 result = YOUR_VALUE electricity × YOUR_VALUE grid-average emission factor
market-based Scope 2 result = YOUR_VALUE electricity × YOUR_VALUE contractual-instrument emission factor
Record the units and source of every factor, and do not mix methods within one result. Keep manufacturing emissions outside this Scope 2 worksheet unless you have a separate current method for calculating them.
Where used-GPU inspections commonly go wrong
Stop the acceptance process when:
- The reported serial does not match the physical board label.
- Unsupported fields or unexplained Xid messages remain unresolved.
- Retired pages, row-remapper warnings, physical damage, or connector problems have no documented disposition.
- The card does not fit the recorded slot, clearance, power, or cooling constraints.
- Container verification does not expose the intended GPU.
- The deployment relies on an unresolved software licence or treats SOC reporting as hardware certification.
These steps are compiled from the official documentation linked in the text and should be checked against the current version before use.