Search / Retrieval / Reranking discussion

[cudaline]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about VRAM headroom.

I'm especially interested in mistakes worth avoiding.

[sparrowgrid]

Posting this in case it saves someone else a little time.

following this because I'm in the same boat.

[phasestream53]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.