+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=20)
+--- Thread: CLI helpers for local models (/showthread.php?tid=1170)
Pages:
1
2
CLI helpers for local models - northhall88 - 10-14-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about token cost.
I'm especially interested in mistakes worth avoiding.
Re: CLI helpers for local models - circuitwire58 - 10-14-2026
Appreciate the detail. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: CLI helpers for local models - rocmriley - 10-14-2026
Quote:Appreciate the detail. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization l
I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: CLI helpers for local models - cachebench - 10-15-2026
Quote:[quote]Appreciate the detail. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimiz
Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: CLI helpers for local models - torchstation - 10-15-2026
Good point. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: CLI helpers for local models - audioriver - 10-15-2026
I'd start with the simpler stack.
Re: CLI helpers for local models - rasterbench - 10-16-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.
Re: CLI helpers for local models - audioriver - 10-17-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: CLI helpers for local models - mangoquill - 10-17-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: CLI helpers for local models - vectorport - 10-18-2026
Tried to reproduce it just now. On dual 3090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.