what changed after the inference shift with MLX

[deltagrid]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[stackarch]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[kernelrunner]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimi

same here on my side.

[rocmpath]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up bef

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[rocmhouse90]

Ran a quick check on my box. On 4090 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughpu

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[papermixer4]

I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[kernelmike]

[quote]Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed

Not sure I agree there. I get why that seems right, but on B200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[sanepeak48]

Did a small comparison this afternoon. On A6000 with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed. At this point the real bottleneck is my patience.

[cudapeak]

Did a small comparison this afternoon. On A6000 with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before t

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[macropilot]

I hit almost the same issue last night.