Vector Databases discussion: anyone testing this?

[southspark]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[dockridge]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[byteyard]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.

[routeforge64]

What worked for me was pretty simple. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.