+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: Systems & Builds (https://www.datasculpting.ai/inference/forumdisplay.php?fid=29)
+--- Thread: Threadripper build for inference (/showthread.php?tid=760)
Pages:
1
2
Threadripper build for inference - visionforge - 10-05-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Qwen using LM Studio on B200. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.
Curious if anyone else landed in the same place.
This is mostly for local work rather than public serving.
Re: Threadripper build for inference - batchwire - 10-05-2026
Ran a quick check on my box. On dual 3090 with PyTorch, I saw basically the same behavior.
Re: Threadripper build for inference - vrambandit - 10-06-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: Threadripper build for inference - stackpilot - 10-09-2026
That matches what I was seeing. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: Threadripper build for inference - homelabhenry - 10-09-2026
subscribed, curious where this lands.
Re: Threadripper build for inference - bytegarden73 - 10-09-2026
subscribed, curious where this lands.
Re: Threadripper build for inference - batchwire - 10-09-2026
Thanks, that helps. That matches what I saw on 7900 XTX.
Re: Threadripper build for inference - plainscope50 - 10-09-2026
Quote:Thanks, that helps. That matches what I saw on 7900 XTX.
That matches what I was seeing. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Threadripper build for inference - batchwire - 10-09-2026
I'd start with the simpler stack.
Re: Threadripper build for inference - cozygrid12 - 10-10-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.