RTX 5090 local inference: early impressions

[phasenode]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: DeepSeek, llama.cpp, A6000, with a lot of attention on consumer GPUs.

If someone wants the exact settings I can clean them up and post them.

[rocmriley]

Thought I'd post a small recipe because a lot of threads ask similar things.

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[signaltrack44]

following this because I'm in the same boat.

[vrambandit]

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[fusedhall]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[bytelab]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[batchember]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not gl

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.