+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Agents (https://www.datasculpting.ai/inference/forumdisplay.php?fid=44)
+--- Thread: distill or just quantize with Ollama (/showthread.php?tid=859)
Pages:
1
2
distill or just quantize with Ollama - thetaplane - 10-08-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: vLLM on mini PC, keeping HBM constraints in mind from the start.
Feel free to tear it apart if you see something dumb.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: distill or just quantize with Ollama - thetaplane - 10-09-2026
Did a small comparison this afternoon. On A6000 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: distill or just quantize with Ollama - llmtrack26 - 10-09-2026
same here on my side.
Re: distill or just quantize with Ollama - emberpeak - 10-09-2026
I think the bottleneck is somewhere else. I've seen people blame INT4 when the real issue was much more mundane.
Re: distill or just quantize with Ollama - quiettoken - 10-09-2026
subscribed, curious where this lands.
Re: distill or just quantize with Ollama - benchplease - 10-10-2026
Quote:subscribed, curious where this lands.
Appreciate the detail. The pattern around token cost has been pretty similar for me.
Re: distill or just quantize with Ollama - quiettoken - 10-13-2026
Interesting, I hadn't tried that. The pattern around HBM constraints has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: distill or just quantize with Ollama - amberpatch25 - 10-13-2026
following this because I'm in the same boat. The logs were somehow calmer than I was.
Re: distill or just quantize with Ollama - quiettoken - 10-14-2026
Quote:following this because I'm in the same boat. The logs were somehow calmer than I was.
Interesting, I hadn't tried that. The pattern around quant drift has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: distill or just quantize with Ollama - rocmmixer - 10-16-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.