+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Search / Retrieval / Reranking (https://www.datasculpting.ai/inference/forumdisplay.php?fid=61)
+--- Thread: Search / Retrieval / Reranking discussion on B200 (/showthread.php?tid=596)
Search / Retrieval / Reranking discussion on B200 - cudaworks - 09-29-2026
Did a bit of testing and thought I'd post the result here.
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring prefix caching.
Curious if anyone else landed in the same place.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: Search / Retrieval / Reranking discussion on B200 - signalpeak - 09-30-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: Search / Retrieval / Reranking discussion on B200 - visionyard23 - 09-30-2026
Thanks, that helps. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.