+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Model Merging (https://www.datasculpting.ai/inference/forumdisplay.php?fid=57)
+--- Thread: do you even need a vector DB here (/showthread.php?tid=956)
Pages:
1
2
do you even need a vector DB here - torchsmith25 - 10-10-2026
Did a bit of testing and thought I'd post the result here.
Short version: vLLM behaved well for me, but only after I stopped ignoring embeddings.
Nothing scientific here, just practical notes from a real setup.
Re: do you even need a vector DB here - sparrowhall - 10-10-2026
Tried to reproduce it just now. On dual 3090 with llama.cpp, I saw basically the same behavior.
Re: do you even need a vector DB here - cacheline - 10-10-2026
I'd start with the simpler stack.
Re: do you even need a vector DB here - rocmrunner40 - 10-10-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: do you even need a vector DB here - phasescope - 10-13-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: do you even need a vector DB here - mangopeak72 - 10-13-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.
Re: do you even need a vector DB here - visionforge - 10-15-2026
I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: do you even need a vector DB here - circuitworks90 - 10-15-2026
Quote:I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not gla
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: do you even need a vector DB here - byteyard - 10-15-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.
Re: do you even need a vector DB here - emberpeak - 10-16-2026
Quote:What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.
Not sure I agree there. I've seen people blame prefix caching when the real issue was much more mundane.