search and retrieval tuning tips

[homelabhenry]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Curious whether that lines up with what other people are seeing.

[turbogrid11]

I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[deltastation]

Appreciate the detail. I ended up in about the same place using SGLang.

[bytelab]

Appreciate the detail. I ended up in about the same place using SGLang.

Need a little more detail here. Is this local-only or are you routing anything externally?

[turbogrid11]

Good point. That matches what I saw on A6000.

[bytelab]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[audionode8]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.

[turbogrid11]

I'd start with the simpler stack.

[homelabhenry]

Interesting, I hadn't tried that. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[audionode8]

Good point. The pattern around tail latency has been pretty similar for me.