Help with Vector Databases

[amberplane]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and llama.cpp for Llama. The interesting difference for me has been prefill/decode split.

What would you pick if you had to live with it for a few months? I keep promising myself this is the last rebuild.

[tinycontext]

I'd approach it in a boring order. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[relayforge]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

[benchplease]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[tinycontext]

Thanks, that helps. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[amberhall]

Can you fill in a bit more context? What model, what runtime, and roughly what context size? I keep promising myself this is the last rebuild.

[minthall]

Can you fill in a bit more context? What model, what runtime, and roughly what context size? I keep promising myself this is the last rebuild.

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[relayforge]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding.

[oxidegrove40]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[macronest]

What worked for me was pretty simple. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.