+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: simple local serving recipe on 5090 (/showthread.php?tid=432)
Pages:
1
2
Re: simple local serving recipe on 5090 - cobalttinker - 09-28-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: simple local serving recipe on 5090 - oxidestream - 09-29-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: simple local serving recipe on 5090 - rocmhouse90 - 09-29-2026
Tried to reproduce it just now. On GB200 with Ollama, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: simple local serving recipe on 5090 - llmpeak - 09-30-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.
Re: simple local serving recipe on 5090 - rasterbench - 10-01-2026
worth checking the driver first.
Re: simple local serving recipe on 5090 - mintstation99 - 10-04-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.