reranking still worth it for small sets

[homelabhenry]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[flintriver50]

What worked for me was pretty simple. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rocmmixer]

Ran a quick check on my box. On B200 with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.