+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: General (https://www.datasculpting.ai/inference/forumdisplay.php?fid=25)
+--- Thread: what are you all running this week on 5090 (/showthread.php?tid=1345)
Pages:
1
2
what are you all running this week on 5090 - rocmplane - 10-16-2026
This keeps coming up in conversation so I figured I'd start a thread.
One thing I keep noticing is that people underestimate tail latency until they actually try to run this stuff for more than one afternoon.
Curious whether that lines up with what other people are seeing.
Re: what are you all running this week on 5090 - sanepeak48 - 10-18-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: what are you all running this week on 5090 - cacheplane98 - 10-18-2026
A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: what are you all running this week on 5090 - torchkite - 10-18-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
I'm happy to test one more variant if you want a second data point.
Re: what are you all running this week on 5090 - rocmrunner40 - 10-18-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: what are you all running this week on 5090 - latentscope35 - 10-18-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: what are you all running this week on 5090 - tensorkite - 10-18-2026
Good point. That matches what I saw on A6000.
Re: what are you all running this week on 5090 - quiettoken - 10-18-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: what are you all running this week on 5090 - thetagrid15 - 10-18-2026
A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: what are you all running this week on 5090 - rocmplane - 10-18-2026
My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.