RAG recall vs speed tradeoff on Threadripper box

[localpatch78]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.