tokens/sec tuning ideas

[mlxgarden]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty. The logs were somehow calmer than I was.

[rasterhall]

What exact setup are you running? What model, what runtime, and roughly what context size?

[mangotinker73]

subscribed, curious where this lands.

[edgeriver]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[batchfield55]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[quantshed]

What worked for me was pretty simple. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Turns out 'just one more test' was a lie.

[mintforge]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[tensorfield88]

Appreciate the detail. The pattern around MoE serving has been pretty similar for me.

[llm_wren]

Appreciate the detail. The pattern around MoE serving has been pretty similar for me.

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[emberquill]

Good point. I don't think your conclusion is far off.