best local stack on mac right now

[visionyard23]

I hit almost the same issue last night.

I'm happy to test one more variant if you want a second data point.

[turboyard94]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[mlxgarden]

same here on my side.

[thetaarch43]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[circuitwire58]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.