+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: GPUs (https://www.datasculpting.ai/inference/forumdisplay.php?fid=28)
+--- Thread: RTX 5090 local inference (/showthread.php?tid=592)
Pages:
1
2
RTX 5090 local inference - turbogrid11 - 09-29-2026
Did a bit of testing and thought I'd post the result here.
Short version: vLLM behaved well for me, but only after I stopped ignoring NVFP4.
Happy to post more details if anyone wants them.
Re: RTX 5090 local inference - cudapoint - 09-30-2026
Quote:Did a bit of testing and thought I'd post the result here.
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: RTX 5090 local inference - embersmith - 09-30-2026
That matches what I was seeing. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: RTX 5090 local inference - cudapoint - 09-30-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: RTX 5090 local inference - batchwire - 10-03-2026
Appreciate the detail. The pattern around speculative decoding has been pretty similar for me.
Re: RTX 5090 local inference - routeforge64 - 10-03-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
Re: RTX 5090 local inference - signaltrack44 - 10-03-2026
subscribed, curious where this lands.
Re: RTX 5090 local inference - nanobrook - 10-03-2026
I'd approach it in a boring order. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time. The logs were somehow calmer than I was.
Re: RTX 5090 local inference - rocmriley - 10-03-2026
I don't think that's the whole story. I get why that seems right, but on B200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: RTX 5090 local inference - paperrunner - 10-03-2026
following this because I'm in the same boat.