Help with Search / Retrieval / Reranking with LM Studio

[dockriver]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it. I keep promising myself this is the last rebuild.

[rasterpoint]

Spent some time on this today and wanted to compare notes.

Interesting, I hadn't tried that. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[labline]

Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[modgarden]

Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[quietarch]

Good point. The pattern around tool calling has been pretty similar for me.

[quantshed]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[quietarch]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[kernelarch]

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[tinycontext]

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survi

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[edgecoast]

[quote]I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the proble

I'd start with the simpler stack.

I'm happy to test one more variant if you want a second data point.