---
title: "Docker vs native install for local LLMs: a working playbook (2) (2)"
url: https://memory.wiki/xLy_sk23
updated: 2026-11-21T21:50:19.172Z
source: "memory.wiki"
---
# Docker vs native install for local LLMs: a working playbook (2) (2)

## Context for "Docker vs native install for local LLMs"

Docker for isolation; native when GPU passthrough fights you.  Software is the part of a local AI stack that changes fastest. A setup that worked last month can break after a driver update, a runtime upgrade, or a change in how a model is packaged. A short, written procedure keeps those changes boring.

## A minimal procedure

1. Note the operating system, GPU driver, runtime version and model file before changing anything.
2. Make one change: a new runtime, a new quantisation, a new context length, or a new model.
3. Run the same ten to twenty prompts you always use, in the languages your users actually write in.
4. Compare answers and resource use against the previous run.
5. Keep or roll back, and record the decision with a date.

## Installation choices

Native installs are simplest on a single machine and avoid GPU passthrough issues. Containers help when several services share a host or when you need reproducible deployments; mount model volumes rather than baking multi-gigabyte weights into images. On Windows, WSL2 works well but its virtual disk can grow quietly, so watch free space. On macOS, decide whether you will use llama.cpp builds or MLX and keep the directory layout tidy.

## Settings worth documenting

- Default temperature and system prompt for the team, with a reason for any exception.
- Context length and number of parallel slots per model.
- Which features are enabled: tool calls, speculative decoding, vision input.
- Where logs go and how long they are kept.

## When something goes wrong

Collect the exact error text, the command or UI action that triggered it, and resource use at that moment. Many reports turn out to be out-of-memory conditions, a wrong prompt template, or a partially downloaded file. Search a shared failure wiki before posting; add the fix once you find it.

## Checklist for a new machine

- Install the GPU driver first and confirm the card is visible to the system.
- Install one runtime only, and verify it can see the GPU.
- Pull one small model, run a short prompt, and note memory use.
- Move the model directory to a disk with enough free space and update the path.
- Add a second model only after the first one works reliably.
- Write the steps you followed into a shared document while they are fresh.

This order makes problems easy to isolate. When several tools and models are installed at once, it becomes much harder to tell which change caused a failure.

Further reading from MicroAI One: [VRAM and model-size table](https://microaione.com/match/vram-table/), and [local AI tutorials](https://microaione.com/tutorials/).

## Bottom line

Local AI rewards patience and record keeping more than raw spending. Decide what the system must do, measure it on your own hardware and documents, write down every change with a date and an owner, and review the setup after each significant upgrade. Small, documented steps are easier to debug, easier to hand over and easier to defend when someone asks why a particular choice was made. When in doubt, test on a smaller scale first and expand only when the evidence supports it.

## Quick reminders

- Cap concurrent sessions before lowering quantisation.
- Photograph cable routing before closing the case.
- Read the model licence before commercial use.
- Keep a printed fallback for offline demos.

*Published by [MicroAI One (本地AI 1号网)](https://microaione.com/), a Shenzhen team working on local LLM hardware, software and deployment.*

---

## Summary
Local LLM setups require consistent record keeping and incremental testing to manage the frequent updates and potential hardware conflicts inherent in the technology. Documenting every configuration change and installation step simplifies debugging and ensures reproducible deployments.

## Themes
- local LLM deployment
- systematic debugging procedures
- hardware software maintenance
- reproducible AI environments

## Key takeaways
- Native installs are preferred for simplicity, while Docker is better for reproducible deployments and multi-service environments.
- Mount model volumes externally rather than baking large weights directly into Docker images.
- A structured testing procedure involves documenting current versions, making a single change, and comparing performance against a baseline.
- New machine setups should prioritize GPU driver verification and single runtime installation before scaling model counts.
- Documenting default settings like temperature and system prompts ensures consistency across team environments.

## Insights
- The rapid evolution of local AI software makes documentation more critical than hardware specifications for long term stability.
- Isolating variables by changing only one component at a time is the most effective way to debug complex AI stacks.
- Resource management issues like disk space in WSL2 and VRAM limits are often mistaken for software bugs.

## Open questions / gaps
- What specific strategies are recommended for managing GPU passthrough conflicts when Docker is strictly required?

