+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Search / Retrieval / Reranking (https://www.datasculpting.ai/inference/forumdisplay.php?fid=61)
+--- Thread: Search / Retrieval / Reranking discussion with vLLM (/showthread.php?tid=1272)
Search / Retrieval / Reranking discussion with vLLM - visionforge - 10-15-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Phi, SGLang, A6000, with a lot of attention on token cost.
If someone wants the exact settings I can clean them up and post them.
This is mostly for local work rather than public serving.
Re: Search / Retrieval / Reranking discussion with vLLM - promptspark58 - 10-17-2026
That matches what I was seeing. The pattern around continuous batching has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: Search / Retrieval / Reranking discussion with vLLM - cacheplane98 - 10-17-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. The logs were somehow calmer than I was.