Search / Retrieval / Reranking discussion

[mellowwire98]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[sparrowworks88]

Need a little more detail here. What model, what runtime, and roughly what context size?

[cudacrate]

Maybe, but I had the opposite result. I've seen people blame batch size when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[tensorport]

Maybe, but I had the opposite result. I've seen people blame batch size when the real issue was much more mundane. Worth testing the simpler explanati

Appreciate the detail. I don't think your conclusion is far off.

[southtinker]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.