+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: what changed after the inference shift with MLX (/showthread.php?tid=1483)
Pages:
1
2
Re: what changed after the inference shift with MLX - bytestack - 10-19-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.
Re: what changed after the inference shift with MLX - localpatch - 10-19-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: what changed after the inference shift with MLX - sparrowshed - 10-19-2026
Appreciate the detail. I ended up in about the same place using vLLM.
what changed after the inference shift with MLX - turboscope5 - 10-19-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.
Wondering if this is just where the whole space is heading now.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: what changed after the inference shift with MLX - kernelrunner - 10-19-2026
Ran a quick check on my box. On 4090 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
I'm happy to test one more variant if you want a second data point.