CLI helpers for local models

[modgarden]

I'm trying to sanity-check my setup.

My current setup is LM Studio with A6000, and I keep running into VRAM headroom.

Not looking for the perfect answer, mostly what worked in real use.

This is mostly for local work rather than public serving.

[amberpatch25]

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmriley]

I tested this tonight. On 4090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.