CLI helpers for local models

[northhall88]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about token cost.

I'm especially interested in mistakes worth avoiding.

[circuitwire58]

Appreciate the detail. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmriley]

Appreciate the detail. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimization l

I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cachebench]

[quote]Appreciate the detail. That matches what I saw on GB200. The biggest change for me was simplifying the stack instead of adding one more optimiz

Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[torchstation]

Good point. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.

[audioriver]

I'd start with the simpler stack.

[rasterbench]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.

[audioriver]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[mangoquill]

What exact setup are you running? What model, what runtime, and roughly what context size?

[vectorport]

Tried to reproduce it just now. On dual 3090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.