Vector Databases discussion: what are you seeing?

[tinycontext]

Has anyone else run into this?

My current setup is llama.cpp with 4090, and I keep running into consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.

The part that surprised me was how much the boring details mattered more than the model choice.

[syshouse]

Has anyone else run into this?

I hit almost the same issue last night.

[deepseeked]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?