what changed after the inference shift with MLX

[bytestack]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[localpatch]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[sparrowshed]

Appreciate the detail. I ended up in about the same place using vLLM.

[turboscope5]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.

[kernelrunner]

Ran a quick check on my box. On 4090 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

I'm happy to test one more variant if you want a second data point.