desk setup thread on 5090

[macropilot]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of SGLang and Threadripper box felt surprisingly usable once I tuned around HBM constraints.

Feel free to tear it apart if you see something dumb.

[rocmmixer]

Thought I'd post a small recipe because a lot of threads ask similar things.

Good point. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[thetatinker77]

A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[turbogrid11]

That matches what I was seeing. That matches what I saw on 7900 XTX.

[sparrowgrid]

A couple questions before I guess too hard. Are you testing this on 5090 or something else?

[kernelpeak]

that fixed it for me too.

[batchyard]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Not glamorous advice, but repeatable baselines save a lot of time.

[bytehall]

same here on my side.

[fusedriver]

same here on my side.

[cudascope]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.