vector DB that stayed out of the way: anyone testing this?

[vrampilot]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.

[vectorcrate98]

worth checking the driver first.

[plainember52]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[phasespark]

I tested this tonight. On 5090 with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.

[signaltrack44]

subscribed, curious where this lands.

[rocmriley]

subscribed, curious where this lands.

Appreciate the detail. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[torchshed66]

Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[northgarden37]

Need a little more detail here. Are you testing this on M3 Max or something else?

[audiodesk]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

I'm happy to test one more variant if you want a second data point.

[deepseeked]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.