vector DB that stayed out of the way on dual 3090

[nanogarden]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with reranking.

Wondering if this is just where the whole space is heading now.

[benchplease]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[paperrunner]

A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece. Turns out 'just one more test' was a lie.

[vramstream]

What exact setup are you running? What model, what runtime, and roughly what context size?

[quietcraft]

I tested this tonight. On B200 with TensorRT-LLM, I saw basically the same behavior. The main thing that moved was latency, not quality.

[sparrowshed]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[phasespark]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[tinycontext]

Did a small comparison this afternoon. On Threadripper box with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.