workflow automation with local models with Ollama

[llm_wren]

Maybe I'm missing something obvious.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling HBM constraints.

If you've tested something similar, what did you keep and what did you stop bothering with?

[llm_wren]

Maybe I'm missing something obvious.

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[bytegarden73]

Thanks, that helps. The pattern around KV cache pressure has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[southden]

Thanks, that helps. The pattern around structured output has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[llm_wren]

Good point. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[phasestream53]

subscribed, curious where this lands.

[labline]

What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetanest]

What exact setup are you running? Are you testing this on 7900 XTX or something else?

[llm_wren]

Thanks, that helps. The pattern around tail latency has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cudaline]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.