hybrid retrieval results for Mixtral

[bytelab]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, llama.cpp, M3 Max, with a lot of attention on tokens/sec.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[homelabhenry]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[mintstream9]

My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[batchsize]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

I'm happy to test one more variant if you want a second data point.

[cacheworks]

Not sure I agree there. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[kernelghost]

What exact setup are you running? Are you testing this on Threadripper box or something else?

[tinycontext]

A couple questions before I guess too hard. Are you testing this on B200 or something else?

[labpath]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[homelabhenry]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

A couple questions before I guess too hard. Are you testing this on 4090 or something else?

[relayforge]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

subscribed, curious where this lands.