Help with Vector Databases

[emberquill]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and PyTorch for Qwen. The interesting difference for me has been KV cache pressure.

What would you pick if you had to live with it for a few months?

[macronest]

Spent some time on this today and wanted to compare notes.

What worked for me was pretty simple. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[amberhall]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.