+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: GPUs (https://www.datasculpting.ai/inference/forumdisplay.php?fid=28)
+--- Thread: RTX 5090 local inference for Llama (/showthread.php?tid=1180)
Pages:
1
2
RTX 5090 local inference for Llama - torchshed - 10-13-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once token cost becomes important.
Interested in tradeoffs more than fan loyalty.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: RTX 5090 local inference for Llama - cudapoint - 10-15-2026
Quote:Spent some time on this today and wanted to compare notes.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.
Re: RTX 5090 local inference for Llama - nanopatch - 10-17-2026
A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: RTX 5090 local inference for Llama - batchwire - 10-18-2026
I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: RTX 5090 local inference for Llama - nanopatch - 10-18-2026
Appreciate the detail. The pattern around consumer GPUs has been pretty similar for me.
Re: RTX 5090 local inference for Llama - rocmriley - 10-19-2026
I hit almost the same issue last night.
Re: RTX 5090 local inference for Llama - rocmriley - 10-19-2026
That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: RTX 5090 local inference for Llama - amberhall - 10-19-2026
I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: RTX 5090 local inference for Llama - dockkite69 - 10-19-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: RTX 5090 local inference for Llama - amberhall - 10-19-2026
I hit almost the same issue last night.