Threadripper build for inference: setup advice

[homelabhenry]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on dual 3090, keeping prefill/decode split in mind from the start.

Maybe useful as a starting point for newer folks.

[latentpeak22]

Can you fill in a bit more context? Is this local-only or are you routing anything externally? I keep promising myself this is the last rebuild.

[batchwire]

Can you fill in a bit more context? Is this local-only or are you routing anything externally? I keep promising myself this is the last rebuild.

Appreciate the detail. That matches what I saw on 5090.

[phasenode]

I think the bottleneck is somewhere else. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

[vectorcrate98]

That matches what I was seeing. That matches what I saw on dual 3090.

[labnest]

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[homelabhenry]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Not glamorous advice, but repeatable baselines save a lot of time.

[macroforge]

Thanks, that helps. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.