what are you all running this week for Phi

[ggufwire22]

Good point. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[tensorkite]

Good point. The pattern around speculative decoding has been pretty similar for me.

[cobaltworks25]

following this because I'm in the same boat.

[marbletrack]

I tested this tonight. On GB200 with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[thetaember38]

I tested this tonight. On GB200 with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collaps

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.