RTX 5090 local inference for Llama

[torchshed]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once token cost becomes important.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[cudapoint]

Spent some time on this today and wanted to compare notes.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.

[nanopatch]

A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[batchwire]

I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[nanopatch]

Appreciate the detail. The pattern around consumer GPUs has been pretty similar for me.

[rocmriley]

I hit almost the same issue last night.

[rocmriley]

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[amberhall]

I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[dockkite69]

Need a little more detail here. Is this local-only or are you routing anything externally?

[amberhall]

I hit almost the same issue last night.