---
title: "Acceptable used-GPU compromises: what we learned"
url: https://memory.wiki/qcXF0Wln
updated: 2026-11-17T21:52:35.857Z
source: "memory.wiki"
---
# Acceptable used-GPU compromises: what we learned

## The question behind "Acceptable used-GPU compromises"

Cosmetic OK; thermal paste neglect not OK.  Most hardware threads about local LLMs jump straight to a card model. A better starting point is the workload: which model families you actually need, the quantisation you can live with, the context length your users paste, and how many people will hit the box at once. Only then does a shopping list make sense.

## Work out the memory budget first

Model weights are only part of the picture. The KV cache grows with context length and concurrent sessions, the runtime keeps its own buffers, and the desktop or display server may already be holding a slice of VRAM before you load anything. Write down free memory after a cold boot, not the number on the box. If the model plus a realistic context does not fit with headroom, it will spill to system RAM and feel slow in a way that no setting fixes. A [hardware reviews](https://microaione.com/hardware/) is useful here because it lets you reason in parameter counts and quant levels instead of marketing tiers.

## Platform matters as much as the card

A good GPU on a weak platform is a common regret. Check:

- PSU capacity and quality, with real headroom for transient spikes, not just the nominal TDP.
- Physical clearance: card length, thickness, and the bend radius of the power cable.
- PCIe slot wiring if you plan to add a second card later.
- Airflow: intake and exhaust paths that still work when the side panel is on and the room is warm.
- System RAM: enough to hold the OS, the inference server, and an embedding model without swapping.

## New, used, or unified memory

Used cards can be good value, but inspect them: missing screws, replaced fans, discoloured connectors and odd thermals under a short load test are reasons to walk away. Apple Silicon machines take a different route: unified memory means the model and the system share one pool, so leave clear headroom for macOS and anything else you run. Mini PCs are fine for small models and kiosks if you accept their thermal limits. Whatever you choose, record serial numbers, invoices and photos on install day.

## A sane buying sequence

1. Write the workload in one paragraph: models, context, users, hours.
2. Map it to memory needs with headroom.
3. Choose the platform (PSU, board, case, RAM), then the GPU.
4. Burn in for at least half an hour while watching temperatures and logs.
5. Run your own small eval set before declaring the build done.

## Questions to ask before you buy

- Will the model you need still fit after the operating system, drivers and runtime take their share?
- How long are the documents people really paste, and how many people paste them at the same time?
- Does the room tolerate the noise and heat of a sustained load during working hours?
- Who replaces a failed fan or PSU, and how long can the team live without the box?
- Is there a clear upgrade path, or will the next step require a new platform?

If any answer is unclear, rent or borrow time on similar hardware first. A week of real use teaches more than a month of reading spec sheets, and it keeps your purchase grounded in your own workload.

See also [custom local-AI builds](https://microaione.com/builds/).

## Quick reminders

- Publish what the assistant will refuse to do.
- Keep the previous model file until the new one passes your eval set.
- Review the setup again after every major model upgrade.
- Measure on your own prompts before quoting any speed.

*Published by [MicroAI One (本地AI 1号网)](https://microaione.com/), a Shenzhen team working on local LLM hardware, software and deployment.*

---

## Summary
When building hardware for local LLMs, you should define your specific workload and memory requirements before selecting a platform and GPU. Prioritize sufficient VRAM headroom for context and system overhead while ensuring your power supply and cooling can handle sustained operational loads.

## Themes
- Local LLM hardware planning
- Used GPU evaluation
- System architecture requirements
- Workload-first deployment

## Key takeaways
- Workload requirements including model family, quantization, and context length should be defined before selecting hardware.
- Total VRAM needs must account for model weights, KV cache, runtime buffers, and OS overhead to avoid system RAM spillover.
- Platform stability depends on PSU transient spike capacity, physical case clearance, and PCIe slot configuration.
- Used hardware requires inspection for signs of neglect such as missing screws, fan replacements, or thermal instability.
- A burn-in period of at least thirty minutes is necessary to verify thermal and operational stability before production use.

## Insights
- VRAM capacity should be calculated based on free memory after a cold boot rather than the manufacturer's nominal box specifications.
- A high-performance GPU is often bottlenecked by inadequate platform support like insufficient PSU headroom or poor thermal management.
- Real-world testing with a small evaluation set is more valuable for performance validation than relying on theoretical spec sheets.

## Open questions / gaps
- What specific metrics or benchmarks constitute a successful evaluation set for different model types?

