search and retrieval tuning tips for Mixtral

[quietarch]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[kernelghost]

Good point. The pattern around tail latency has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[routeforge64]

following this because I'm in the same boat.

[orbitgarden]

My first guess would be this: On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[cacheline]

Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.