RAG recall vs speed tradeoff on Threadripper box

[llm_wren]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[turboarch95]

Need a little more detail here. Are you testing this on B200 or something else?