RAG recall vs speed tradeoff with TensorRT-LLM

[cudapeak]

Did a bit of testing and thought I'd post the result here.

After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Curious if anyone else landed in the same place.

[thetashed24]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.