how much orchestration is too much with Ollama

[llm_wren]

Did a bit of testing and thought I'd post the result here.

After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Nothing scientific here, just practical notes from a real setup.

[quantsupply]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

[torchtrack24]

Interesting, I hadn't tried that. I ended up in about the same place using SGLang.

[routerunner]

What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[cudacrate]

Thanks, that helps. The pattern around workstation thermals has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[phasestream53]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[localpatch78]

I think the bottleneck is somewhere else. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

[llm_wren]

Good point. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[northgrid33]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size? I keep promising myself this is the last rebuild.

[cudacrate]

Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.