VRAM bottleneck on consumer cards for Qwen

[embersmith]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[turboyard94]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not

subscribed, curious where this lands.

[mellowwire98]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals.

[dockkite69]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[paperrunner]

I tested this tonight. On dual 3090 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[torchshed]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but

Maybe, but I had the opposite result. I've seen people blame VRAM headroom when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[rocmriley]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

[rocmriley]

[quote]My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advic

Interesting, I hadn't tried that. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmriley]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[fusedhall]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.