+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: CLI helpers for local models: setup advice (/showthread.php?tid=1211)
CLI helpers for local models: setup advice - cachebench - 10-14-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between LM Studio and PyTorch for Gemma. The interesting difference for me has been MoE serving.
Interested in tradeoffs more than fan loyalty.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: CLI helpers for local models: setup advice - phasetrack - 10-15-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: CLI helpers for local models: setup advice - kernelgrid - 10-15-2026
A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: CLI helpers for local models: setup advice - mlxgarden - 10-18-2026
I don't think that's the whole story. I get why that seems right, but on A6000 the bottleneck felt elsewhere.
Re: CLI helpers for local models: setup advice - latentscope35 - 10-18-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Once you know the plain path works, then add the faster path back in piece by piece. At this point the real bottleneck is my patience.
Re: CLI helpers for local models: setup advice - edgequill - 10-18-2026
Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.