+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Vector Databases (https://www.datasculpting.ai/inference/forumdisplay.php?fid=60)
+--- Thread: Vector Databases discussion with Ollama (/showthread.php?tid=475)
Pages:
1
2
Vector Databases discussion with Ollama - macronest - 09-25-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Phi, llama.cpp, A6000, with a lot of attention on reranking.
If someone wants the exact settings I can clean them up and post them.
Re: Vector Databases discussion with Ollama - tinycontext - 09-25-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: Vector Databases discussion with Ollama - southspark - 09-25-2026
Thanks, that helps. I ended up in about the same place using PyTorch.
Re: Vector Databases discussion with Ollama - deltagrid - 09-25-2026
Interesting, I hadn't tried that. The pattern around tokens/sec has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: Vector Databases discussion with Ollama - fastfield4 - 09-25-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: Vector Databases discussion with Ollama - vramscope - 09-28-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Re: Vector Databases discussion with Ollama - deltagrid - 09-29-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Vector Databases discussion with Ollama - llmarch83 - 09-29-2026
Quote:What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Not glamorous advice,
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: Vector Databases discussion with Ollama - emberquill - 09-30-2026
that fixed it for me too. The logs were somehow calmer than I was.
Re: Vector Databases discussion with Ollama - southspark - 10-01-2026
Good point. I ended up in about the same place using Ollama.