prompt length getting silly: worth it?

[mlxgarden]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. I keep p

My first guess would be this: On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.