little utility you keep installed for Qwen

[relayforge]

Has anyone else run into this?

My current setup is LM Studio with dual 3090, and I keep running into consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

[rasterbench]

Has anyone else run into this?

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[rocmriley]

Interesting, I hadn't tried that. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[mlxgarden]

Thanks, that helps. The pattern around tokens/sec has been pretty similar for me.

[rasterbench]

Thanks, that helps. The pattern around tokens/sec has been pretty similar for me.

A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[tinycontext]

I tested this tonight. On mini PC with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[kernelmike]

subscribed, curious where this lands.

[batchshed]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.

[visionforge]

I'd start with the simpler stack.

[kernelgrid]

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.