+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: RAG (https://www.datasculpting.ai/inference/forumdisplay.php?fid=58)
+--- Thread: search and retrieval tuning tips (/showthread.php?tid=967)
search and retrieval tuning tips - sparrowhall - 10-09-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between MLX and llama.cpp for Mistral. The interesting difference for me has been quant drift.
What would you pick if you had to live with it for a few months?
Re: search and retrieval tuning tips - homelabhenry - 10-11-2026
Appreciate the detail. The pattern around token cost has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable. Turns out 'just one more test' was a lie.
Re: search and retrieval tuning tips - audionode8 - 10-13-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: search and retrieval tuning tips - audiopoint18 - 10-13-2026
subscribed, curious where this lands.
Re: search and retrieval tuning tips - orbitgarden - 10-13-2026
Good point. The pattern around tool calling has been pretty similar for me.
Re: search and retrieval tuning tips - turbogrid11 - 10-14-2026
Quote:Good point. The pattern around tool calling has been pretty similar for me.
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: search and retrieval tuning tips - quantsupply - 10-14-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Re: search and retrieval tuning tips - routecrate - 10-14-2026
Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Good point. I don't think your conclusion is far off.
Re: search and retrieval tuning tips - batchshed - 10-14-2026
Quote:Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.
Re: search and retrieval tuning tips - routeforge64 - 10-17-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.