+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: tokens/sec tuning ideas (/showthread.php?tid=1107)
Pages:
1
2
tokens/sec tuning ideas - mlxgarden - 10-12-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.
Interested in tradeoffs more than fan loyalty. The logs were somehow calmer than I was.
Re: tokens/sec tuning ideas - rasterhall - 10-12-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: tokens/sec tuning ideas - mangotinker73 - 10-12-2026
subscribed, curious where this lands.
Re: tokens/sec tuning ideas - edgeriver - 10-15-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: tokens/sec tuning ideas - batchfield55 - 10-15-2026
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: tokens/sec tuning ideas - quantshed - 10-16-2026
What worked for me was pretty simple. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Turns out 'just one more test' was a lie.
Re: tokens/sec tuning ideas - mintforge - 10-17-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off.
Re: tokens/sec tuning ideas - tensorfield88 - 10-17-2026
Appreciate the detail. The pattern around MoE serving has been pretty similar for me.
Re: tokens/sec tuning ideas - llm_wren - 10-17-2026
Quote:Appreciate the detail. The pattern around MoE serving has been pretty similar for me.
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: tokens/sec tuning ideas - emberquill - 10-19-2026
Good point. I don't think your conclusion is far off.