+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Model Merging (https://www.datasculpting.ai/inference/forumdisplay.php?fid=57)
+--- Thread: vector DB that stayed out of the way: anyone testing this? (/showthread.php?tid=476)
Pages:
1
2
vector DB that stayed out of the way: anyone testing this? - vrampilot - 09-25-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.
Maybe obvious to some of you, but it clicked for me this week.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: vector DB that stayed out of the way: anyone testing this? - vectorcrate98 - 09-25-2026
worth checking the driver first.
Re: vector DB that stayed out of the way: anyone testing this? - plainember52 - 09-25-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: vector DB that stayed out of the way: anyone testing this? - phasespark - 09-28-2026
I tested this tonight. On 5090 with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: vector DB that stayed out of the way: anyone testing this? - signaltrack44 - 09-28-2026
subscribed, curious where this lands.
Re: vector DB that stayed out of the way: anyone testing this? - rocmriley - 09-29-2026
Quote:subscribed, curious where this lands.
Appreciate the detail. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: vector DB that stayed out of the way: anyone testing this? - torchshed66 - 09-29-2026
Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: vector DB that stayed out of the way: anyone testing this? - northgarden37 - 10-02-2026
Need a little more detail here. Are you testing this on M3 Max or something else?
Re: vector DB that stayed out of the way: anyone testing this? - audiodesk - 10-02-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
I'm happy to test one more variant if you want a second data point.
Re: vector DB that stayed out of the way: anyone testing this? - deepseeked - 10-03-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.