benchmarks that map to real use on M3 Max

[llmpeak]

My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[modbench]

Good point. The pattern around long context has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[sanepeak48]

Did a small comparison this afternoon. On Threadripper box with Ollama, I saw basically the same behavior. I expected the model choice to matter more than it did.

[signaltrack44]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[orbitline]

[quote]Did a small comparison this afternoon. On Threadripper box with Ollama, I saw basically the same behavior. I expected the model choice to matte

That matches what I was seeing. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rocmhouse90]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

If you post your exact settings, people here can probably narrow it down pretty fast.