vector DB that stayed out of the way

[cozygrid12]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and GB200.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.

[deltagrid]

That matches what I was seeing. I ended up in about the same place using Ollama. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[torchdesk]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[kernelhouse70]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Not glamorous advice, but repeatable baselines save a lot of time.

[oxiderunner]

that fixed it for me too.

[routeforge64]

Appreciate the detail. I don't think your conclusion is far off.

[edgeriver]

That matches what I was seeing. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.