+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Vector Databases (https://www.datasculpting.ai/inference/forumdisplay.php?fid=60)
+--- Thread: Help with Vector Databases (/showthread.php?tid=1277)
Help with Vector Databases - tinycontext - 10-15-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.
Nothing scientific here, just practical notes from a real setup.
This is mostly for local work rather than public serving.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: Help with Vector Databases - deltagrid - 10-17-2026
I don't think that's the whole story. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: Help with Vector Databases - emberquill - 10-17-2026
What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: Help with Vector Databases - minthall - 10-17-2026
I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: Help with Vector Databases - byteyard - 10-17-2026
My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: Help with Vector Databases - quietsupply11 - 10-17-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: Help with Vector Databases - fastfield4 - 10-17-2026
What worked for me was pretty simple. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Help with Vector Databases - tinycontext - 10-19-2026
worth checking the driver first.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: Help with Vector Databases - deltagrid - 10-19-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: Help with Vector Databases - orbitbench6 - 10-19-2026
I hit almost the same issue last night.