comparing models without fooling yourself on Threadripper box

[benchplease]

Thanks, that helps. That matches what I saw on dual 3090.

[sparrowworks88]

Tried to reproduce it just now. On mini PC with Ollama, I saw basically the same behavior.

[batchsize]

Good point. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

I'm happy to test one more variant if you want a second data point.

[kernelghost]

Good point. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflo

Appreciate the detail. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[deltastation]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

[plainline]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.