+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: Tensor cores actually helping here for Gemma (/showthread.php?tid=877)
Tensor cores actually helping here for Gemma - phaseport36 - 10-07-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Qwen, LM Studio, 5090, with a lot of attention on router policy.
Maybe useful as a starting point for newer folks.
Re: Tensor cores actually helping here for Gemma - orbitbench6 - 10-07-2026
Interesting, I hadn't tried that. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Tensor cores actually helping here for Gemma - vectorember - 10-07-2026
Quote:Interesting, I hadn't tried that. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding
Good point. I ended up in about the same place using llama.cpp.
Re: Tensor cores actually helping here for Gemma - orbitbench6 - 10-07-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: Tensor cores actually helping here for Gemma - dockember - 10-10-2026
I'd start with the simpler stack.
Re: Tensor cores actually helping here for Gemma - thetapath - 10-11-2026
Quote:I'd start with the simpler stack.
following this because I'm in the same boat.
Re: Tensor cores actually helping here for Gemma - phaseport36 - 10-13-2026
Appreciate the detail. The pattern around workstation thermals has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
I'm happy to test one more variant if you want a second data point.
Re: Tensor cores actually helping here for Gemma - marbletrack - 10-14-2026
Interesting, I hadn't tried that. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
If you post your exact settings, people here can probably narrow it down pretty fast.