+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Search / Retrieval / Reranking (https://www.datasculpting.ai/inference/forumdisplay.php?fid=61)
+--- Thread: Search / Retrieval / Reranking discussion with Ollama (/showthread.php?tid=143)
Pages:
1
2
Search / Retrieval / Reranking discussion with Ollama - kernelhouse70 - 09-06-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.
Curious whether that lines up with what other people are seeing.
This is mostly for local work rather than public serving.
Re: Search / Retrieval / Reranking discussion with Ollama - deltapoint17 - 09-07-2026
A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: Search / Retrieval / Reranking discussion with Ollama - torchsupply99 - 09-07-2026
Quote:A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Can you fill in a bit more context? Are you testing this on 5090 or something else?
Re: Search / Retrieval / Reranking discussion with Ollama - cacheline - 09-07-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: Search / Retrieval / Reranking discussion with Ollama - oxidemixer - 09-10-2026
Quote:Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up
subscribed, curious where this lands.
Re: Search / Retrieval / Reranking discussion with Ollama - visionforge - 09-10-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: Search / Retrieval / Reranking discussion with Ollama - audioforge - 09-10-2026
I don't think that's the whole story. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps. At this point the real bottleneck is my patience.
Re: Search / Retrieval / Reranking discussion with Ollama - llm_wren - 09-10-2026
I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Search / Retrieval / Reranking discussion with Ollama - audiopath66 - 09-11-2026
Quote:I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not g
A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?
Re: Search / Retrieval / Reranking discussion with Ollama - northcraft90 - 09-11-2026
Interesting, I hadn't tried that. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.