Search / Retrieval / Reranking discussion on B200

[cudaworks]

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring prefix caching.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.

[signalpeak]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[visionyard23]

Thanks, that helps. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.