do you even need a vector DB here

[emberpeak]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.

Not sure I agree there. I've seen people blame prefix caching when the real issue was much more mundane.