Threadripper build for inference

[nanobrook]

I figured I'd ask here before changing everything.

My current setup is MLX with GB200, and I keep running into embeddings.

If you've tested something similar, what did you keep and what did you stop bothering with?

[kernelhouse70]

I figured I'd ask here before changing everything.

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Not glamorous advice, but repeatable baselines save a lot of time.

[stackpilot]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you know the plain path works, then add the faster path back in piece by piece.

[cozygrid12]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[tensorworks]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[sanevault]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time. I keep promising myself this is the last rebuild.

[plainscope50]

That matches what I was seeing. The pattern around embeddings has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[batchyard]

Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tensorport]

Thanks, that helps. I ended up in about the same place using llama.cpp.

[vectorcrate98]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.