vector DB that stayed out of the way with vLLM

[dockember]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

[homelabhenry]

I'd start with the simpler stack.

[thetafield]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[orbitbench6]

A couple questions before I guess too hard. Are you testing this on 4090 or something else?

[mlxgarden]

that fixed it for me too.

[fastscope]

that fixed it for me too.

I hit almost the same issue last night.

[ggufvault]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[stackpilot]

That matches what I was seeing. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tensorfield88]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[sparrowkite53]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.