+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Vector Databases (https://www.datasculpting.ai/inference/forumdisplay.php?fid=60)
+--- Thread: Help with Vector Databases (/showthread.php?tid=898)
Pages:
1
2
Help with Vector Databases - amberplane - 10-08-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between LM Studio and llama.cpp for Llama. The interesting difference for me has been prefill/decode split.
What would you pick if you had to live with it for a few months? I keep promising myself this is the last rebuild.
Re: Help with Vector Databases - tinycontext - 10-09-2026
I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Help with Vector Databases - relayforge - 10-10-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal.
Re: Help with Vector Databases - benchplease - 10-10-2026
Quote:I tested this tonight. My result was mixed: first run looked bad, second run looked normal.
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
Re: Help with Vector Databases - tinycontext - 10-10-2026
Thanks, that helps. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: Help with Vector Databases - amberhall - 10-10-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size? I keep promising myself this is the last rebuild.
Re: Help with Vector Databases - minthall - 10-11-2026
Quote:Can you fill in a bit more context? What model, what runtime, and roughly what context size? I keep promising myself this is the last rebuild.
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: Help with Vector Databases - relayforge - 10-12-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding.
Re: Help with Vector Databases - oxidegrove40 - 10-12-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: Help with Vector Databases - macronest - 10-12-2026
What worked for me was pretty simple. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.