CLI helpers for local models: setup advice

[cachebench]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and PyTorch for Gemma. The interesting difference for me has been MoE serving.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[phasetrack]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[kernelgrid]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[mlxgarden]

I don't think that's the whole story. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

[latentscope35]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Once you know the plain path works, then add the faster path back in piece by piece. At this point the real bottleneck is my patience.

[edgequill]

Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.