where open models feel surprisingly good for Phi

[thetapath]

I'm trying to sanity-check my setup.

I'm mostly working with FLUX in Ollama on B200. The part I'm trying to understand is KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.

[mellowwire98]

My first guess would be this: On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[macrostack69]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[rocmhouse90]

Good point. The pattern around token cost has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[promptpoint]

That matches what I was seeing. I ended up in about the same place using LM Studio. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[sanepeak48]

What exact setup are you running? Are you testing this on 7900 XTX or something else?

[embershed99]

Did a small comparison this afternoon. On M3 Max with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.

[batchsize]

Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[kernelarch]

I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[paperrunner]

A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.