Threadripper build for inference

[visionforge]

Did a bit of testing and thought I'd post the result here.

I tried this with Qwen using LM Studio on B200. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Curious if anyone else landed in the same place.

This is mostly for local work rather than public serving.

[batchwire]

Ran a quick check on my box. On dual 3090 with PyTorch, I saw basically the same behavior.

[vrambandit]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[stackpilot]

That matches what I was seeing. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[homelabhenry]

subscribed, curious where this lands.

[bytegarden73]

subscribed, curious where this lands.

[batchwire]

Thanks, that helps. That matches what I saw on 7900 XTX.

[plainscope50]

Thanks, that helps. That matches what I saw on 7900 XTX.

That matches what I was seeing. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchwire]

I'd start with the simpler stack.

[cozygrid12]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.