Tensor cores actually helping here for Gemma

[phaseport36]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Qwen, LM Studio, 5090, with a lot of attention on router policy.

Maybe useful as a starting point for newer folks.

[orbitbench6]

Interesting, I hadn't tried that. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vectorember]

Interesting, I hadn't tried that. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding

Good point. I ended up in about the same place using llama.cpp.

[orbitbench6]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[dockember]

I'd start with the simpler stack.

[thetapath]

I'd start with the simpler stack.

following this because I'm in the same boat.

[phaseport36]

Appreciate the detail. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

I'm happy to test one more variant if you want a second data point.

[marbletrack]

Interesting, I hadn't tried that. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.