+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: Systems & Builds (https://www.datasculpting.ai/inference/forumdisplay.php?fid=29)
+--- Thread: Threadripper build for inference (/showthread.php?tid=663)
Pages:
1
2
Threadripper build for inference - nanobrook - 10-03-2026
I figured I'd ask here before changing everything.
My current setup is MLX with GB200, and I keep running into embeddings.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: Threadripper build for inference - kernelhouse70 - 10-03-2026
Quote:I figured I'd ask here before changing everything.
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Threadripper build for inference - stackpilot - 10-03-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Threadripper build for inference - cozygrid12 - 10-04-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Threadripper build for inference - tensorworks - 10-05-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: Threadripper build for inference - sanevault - 10-08-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time. I keep promising myself this is the last rebuild.
Re: Threadripper build for inference - plainscope50 - 10-08-2026
That matches what I was seeing. The pattern around embeddings has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: Threadripper build for inference - batchyard - 10-08-2026
Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: Threadripper build for inference - tensorport - 10-08-2026
Thanks, that helps. I ended up in about the same place using llama.cpp.
Re: Threadripper build for inference - vectorcrate98 - 10-08-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.