hybrid retrieval results

[sparrowbrook]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of MLX and Threadripper box felt surprisingly usable once I tuned around batch size.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.

[turbogrid11]

My first guess would be this: If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[homelabhenry]

What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[kernelghost]

What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llmtrack]

I think the bottleneck is somewhere else. I've seen people blame quant drift when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[nanogarden]

I think the bottleneck is somewhere else. I've seen people blame quant drift when the real issue was much more mundane. Worth testing the simpler expl

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[bytelab]

Appreciate the detail. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[homelabhenry]

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[sanepeak48]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tinycontext]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.