recipes for a good local stack on B200

[modgarden]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

[vramstream]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

[llmpeak]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.

[cudaworks]

Interesting, I hadn't tried that. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[rocmriley]

I'd start with the simpler stack.

[deltagrid]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. Not glamorous advice, but repeatable baselines save a lot of time.

[audioriver]

I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[rocmcrate26]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[cozysmith]

Thanks, that helps. I don't think your conclusion is far off.

[mintstation99]

I hit almost the same issue last night.