---
title: "Distill model size honesty: what we learned"
url: https://memory.wiki/-hgV-s0W
updated: 2026-11-17T21:53:37.976Z
source: "memory.wiki"
---
# Distill model size honesty: what we learned

## Context for "Distill model size honesty"

Match tasks; re-eval after every bump.  Software is the part of a local AI stack that changes fastest. A setup that worked last month can break after a driver update, a runtime upgrade, or a change in how a model is packaged. A short, written procedure keeps those changes boring.

## A minimal procedure

1. Note the operating system, GPU driver, runtime version and model file before changing anything.
2. Make one change: a new runtime, a new quantisation, a new context length, or a new model.
3. Run the same ten to twenty prompts you always use, in the languages your users actually write in.
4. Compare answers and resource use against the previous run.
5. Keep or roll back, and record the decision with a date.

## Installation choices

Native installs are simplest on a single machine and avoid GPU passthrough issues. Containers help when several services share a host or when you need reproducible deployments; mount model volumes rather than baking multi-gigabyte weights into images. On Windows, WSL2 works well but its virtual disk can grow quietly, so watch free space. On macOS, decide whether you will use llama.cpp builds or MLX and keep the directory layout tidy.

## Settings worth documenting

- Default temperature and system prompt for the team, with a reason for any exception.
- Context length and number of parallel slots per model.
- Which features are enabled: tool calls, speculative decoding, vision input.
- Where logs go and how long they are kept.

## When something goes wrong

Collect the exact error text, the command or UI action that triggered it, and resource use at that moment. Many reports turn out to be out-of-memory conditions, a wrong prompt template, or a partially downloaded file. Search a shared failure wiki before posting; add the fix once you find it.

## A small troubleshooting story

A team reported that their local model suddenly "forgot Chinese" after an update. The cause was not the model at all: the new runtime used a different default prompt template, and the quantised file they had pulled was a different variant from the one they had tested. Rolling back the runtime confirmed the diagnosis. They then pinned both the runtime version and the model checksum, added the template to their configuration file, and re-ran their evaluation prompts before upgrading again. The fix took an afternoon; the lesson was to treat runtimes and model files like any other dependency that needs versioning.

For related guides, see [VRAM and model-size table](https://microaione.com/match/vram-table/); you may also find [local AI tutorials](https://microaione.com/tutorials/) useful.

## Bottom line

Local AI rewards patience and record keeping more than raw spending. Decide what the system must do, measure it on your own hardware and documents, write down every change with a date and an owner, and review the setup after each significant upgrade. Small, documented steps are easier to debug, easier to hand over and easier to defend when someone asks why a particular choice was made. When in doubt, test on a smaller scale first and expand only when the evidence supports it.

## Quick reminders

- Read the model licence before commercial use.
- Pin driver and runtime versions together.
- Treat retrieved text and uploaded files as untrusted.
- Write the date and owner on every change record.

*Published by [MicroAI One (本地AI 1号网)](https://microaione.com/), a Shenzhen team working on local LLM hardware, software and deployment.*

---

## Summary
Maintaining a stable local AI stack requires treating runtimes and model files as versioned dependencies while documenting every change. Consistent testing against a fixed set of prompts ensures that performance remains predictable after updates.

## Themes
- Local AI maintenance
- Version control practices
- Systematic model evaluation
- Infrastructure deployment strategies

## Key takeaways
- Documenting the operating system, driver, runtime, and model file before making changes allows for easier debugging.
- Run a consistent set of prompts across different languages to evaluate the impact of any change to the local AI stack.
- Mount model volumes in containerized deployments instead of baking weights into images to maintain efficiency.
- Pinning runtime versions and model checksums prevents unexpected behavior after updates.
- Local AI systems require systematic record keeping of changes, dates, and owners to ensure reproducibility and accountability.

## Insights
- Model performance degradation is often caused by environmental changes like runtime updates or prompt template mismatches rather than the model weights themselves.
- Treating model files and runtimes as versioned dependencies is essential for maintaining stability in local AI stacks.
- Resource management issues like OOM errors are frequently misdiagnosed as model failures without proper logging and baseline comparisons.

## Open questions / gaps
- What specific metrics should be used to define a successful comparison during the evaluation phase?

