+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: RAG (https://www.datasculpting.ai/inference/forumdisplay.php?fid=58)
+--- Thread: search and retrieval tuning tips for Mixtral (/showthread.php?tid=81)
Pages:
1
2
search and retrieval tuning tips for Mixtral - modbench - 08-30-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: llama.cpp on Threadripper box, keeping prefix caching in mind from the start.
Maybe useful as a starting point for newer folks.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: search and retrieval tuning tips for Mixtral - deltastation - 08-31-2026
subscribed, curious where this lands.
Re: search and retrieval tuning tips for Mixtral - sparrowbrook - 09-01-2026
I don't think that's the whole story. I've seen people blame structured output when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: search and retrieval tuning tips for Mixtral - sanepeak48 - 09-01-2026
I hit almost the same issue last night.
Re: search and retrieval tuning tips for Mixtral - turboyard94 - 09-01-2026
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: search and retrieval tuning tips for Mixtral - homelabhenry - 09-01-2026
Appreciate the detail. The pattern around tool calling has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: search and retrieval tuning tips for Mixtral - tensorfield88 - 09-01-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal.
Re: search and retrieval tuning tips for Mixtral - macropilot - 09-02-2026
Thanks, that helps. The pattern around KV cache pressure has been pretty similar for me.
Re: search and retrieval tuning tips for Mixtral - sanepeak48 - 09-02-2026
that fixed it for me too.
Re: search and retrieval tuning tips for Mixtral - cacheworks - 09-02-2026
Tried to reproduce it just now. On M3 Max with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.