+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: RAG (https://www.datasculpting.ai/inference/forumdisplay.php?fid=58)
+--- Thread: search and retrieval tuning tips (/showthread.php?tid=1155)
search and retrieval tuning tips - homelabhenry - 10-13-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like prefix caching decides everything downstream.
Curious whether that lines up with what other people are seeing.
Re: search and retrieval tuning tips - turbogrid11 - 10-14-2026
I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: search and retrieval tuning tips - deltastation - 10-14-2026
Appreciate the detail. I ended up in about the same place using SGLang.
Re: search and retrieval tuning tips - bytelab - 10-17-2026
Quote:Appreciate the detail. I ended up in about the same place using SGLang.
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: search and retrieval tuning tips - turbogrid11 - 10-17-2026
Good point. That matches what I saw on A6000.
Re: search and retrieval tuning tips - bytelab - 10-17-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: search and retrieval tuning tips - audionode8 - 10-17-2026
A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: search and retrieval tuning tips - turbogrid11 - 10-17-2026
I'd start with the simpler stack.
Re: search and retrieval tuning tips - homelabhenry - 10-18-2026
Interesting, I hadn't tried that. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: search and retrieval tuning tips - audionode8 - 10-18-2026
Good point. The pattern around tail latency has been pretty similar for me.