+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Model Merging (https://www.datasculpting.ai/inference/forumdisplay.php?fid=57)
+--- Thread: vector DB that stayed out of the way (/showthread.php?tid=724)
Pages:
1
2
vector DB that stayed out of the way - quantyard - 10-03-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about structured output.
Would rather learn from your scars than collect my own all at once.
Re: vector DB that stayed out of the way - bytestack - 10-04-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.
Re: vector DB that stayed out of the way - torchdesk - 10-04-2026
Maybe, but I had the opposite result. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: vector DB that stayed out of the way - benchplease - 10-04-2026
that fixed it for me too.
Re: vector DB that stayed out of the way - vramarch - 10-04-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: vector DB that stayed out of the way - sparrowshed - 10-04-2026
Quote:I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not qual
Interesting, I hadn't tried that. That matches what I saw on A6000.
Re: vector DB that stayed out of the way - ggufvault - 10-05-2026
That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: vector DB that stayed out of the way - northcraft90 - 10-05-2026
Interesting, I hadn't tried that. That matches what I saw on mini PC. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: vector DB that stayed out of the way - bytehall - 10-08-2026
same here on my side.
Re: vector DB that stayed out of the way - mangopeak72 - 10-08-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.