Help with Vector Databases with vLLM

[tinycontext]

Did a bit of testing and thought I'd post the result here.

After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Nothing scientific here, just practical notes from a real setup.

The part that surprised me was how much the boring details mattered more than the model choice.

[fastfield4]

Appreciate the detail. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[emberquill]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

[minthall]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[signaltrack44]

Thanks, that helps. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[amberhall]

I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[tinycontext]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[fastfield4]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

I'm happy to test one more variant if you want a second data point.