Search / Retrieval / Reranking discussion with Ollama

[kernelhouse70]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with speculative decoding.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.

[deltapoint17]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[torchsupply99]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

Can you fill in a bit more context? Are you testing this on 5090 or something else?

[cacheline]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[oxidemixer]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up

subscribed, curious where this lands.

[visionforge]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[audioforge]

I don't think that's the whole story. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps. At this point the real bottleneck is my patience.

[llm_wren]

I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[audiopath66]

I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not g

A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?

[northcraft90]

Interesting, I hadn't tried that. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.