best local runtime for mixed models

[vectorember]

Did a bit of testing and thought I'd post the result here.

After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around token cost.

Nothing scientific here, just practical notes from a real setup.

[edgeriver]

Did a bit of testing and thought I'd post the result here.

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[ggufwire22]

That matches what I was seeing. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[latentpeak22]

That matches what I was seeing. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more opti

I'd start with the simpler stack.

[vectorcrate98]

Good point. The pattern around prefill/decode split has been pretty similar for me.

[dockridge]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[prompttinker44]

That matches what I was seeing. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.