simple local serving recipe on 5090

[cobalttinker]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[oxidestream]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[rocmhouse90]

Tried to reproduce it just now. On GB200 with Ollama, I saw basically the same behavior. The main thing that moved was latency, not quality.

[llmpeak]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.

[rasterbench]

worth checking the driver first.

[mintstation99]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.