Vector Databases discussion with Ollama

[macronest]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, llama.cpp, A6000, with a lot of attention on reranking.

If someone wants the exact settings I can clean them up and post them.

[tinycontext]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

If you post your exact settings, people here can probably narrow it down pretty fast.

[southspark]

Thanks, that helps. I ended up in about the same place using PyTorch.

[deltagrid]

Interesting, I hadn't tried that. The pattern around tokens/sec has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[fastfield4]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[vramscope]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[deltagrid]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[llmarch83]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Not glamorous advice,

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[emberquill]

that fixed it for me too. The logs were somehow calmer than I was.

[southspark]

Good point. I ended up in about the same place using Ollama.