Help with Search / Retrieval / Reranking

[rocmrunner40]

I'd start with the simpler stack.

Can you fill in a bit more context? Are you testing this on A6000 or something else?

[phasespark]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[cozygrid12]

Good point. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[phasescope]

Need a little more detail here. Are you testing this on GB200 or something else?

[plainworks]

Good point. I don't think your conclusion is far off.

[batchcraft]

same here on my side.