distill or just quantize with Ollama

[thetaplane]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: vLLM on mini PC, keeping HBM constraints in mind from the start.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.