+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Model Merging (https://www.datasculpting.ai/inference/forumdisplay.php?fid=57)
+--- Thread: vector DB that stayed out of the way with vLLM (/showthread.php?tid=1212)
Pages:
1
2
Re: vector DB that stayed out of the way with vLLM - vramscope - 10-18-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: vector DB that stayed out of the way with vLLM - vramport - 10-18-2026
Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: vector DB that stayed out of the way with vLLM - vramport - 10-18-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: vector DB that stayed out of the way with vLLM - torchsupply99 - 10-19-2026
What worked for me was pretty simple. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: vector DB that stayed out of the way with vLLM - batchsize - 10-19-2026
Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
I'm happy to test one more variant if you want a second data point.