Help with Vector Databases

[tinycontext]

Did a bit of testing and thought I'd post the result here.

After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[deltagrid]

I don't think that's the whole story. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[emberquill]

What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[minthall]

I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[byteyard]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

[quietsupply11]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[fastfield4]

What worked for me was pretty simple. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[tinycontext]

worth checking the driver first.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[deltagrid]

Need a little more detail here. What model, what runtime, and roughly what context size?

[orbitbench6]

I hit almost the same issue last night.