Search / Retrieval / Reranking discussion with vLLM

[visionforge]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, SGLang, A6000, with a lot of attention on token cost.

If someone wants the exact settings I can clean them up and post them.

This is mostly for local work rather than public serving.

[promptspark58]

That matches what I was seeing. The pattern around continuous batching has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cacheplane98]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. The logs were somehow calmer than I was.