search and retrieval tuning tips for Mixtral

[modbench]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: llama.cpp on Threadripper box, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[deltastation]

subscribed, curious where this lands.

[sparrowbrook]

I don't think that's the whole story. I've seen people blame structured output when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[sanepeak48]

I hit almost the same issue last night.

[turboyard94]

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[homelabhenry]

Appreciate the detail. The pattern around tool calling has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tensorfield88]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

[macropilot]

Thanks, that helps. The pattern around KV cache pressure has been pretty similar for me.

[sanepeak48]

that fixed it for me too.

[cacheworks]

Tried to reproduce it just now. On M3 Max with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.