+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: CPU-only fallback performance for Gemma (/showthread.php?tid=129)
Pages:
1
2
CPU-only fallback performance for Gemma - batchsize - 09-04-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.
Curious whether that lines up with what other people are seeing.
This is mostly for local work rather than public serving.
Re: CPU-only fallback performance for Gemma - localpatch78 - 09-05-2026
That matches what I was seeing. The pattern around INT4 has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
I'm happy to test one more variant if you want a second data point.
Re: CPU-only fallback performance for Gemma - fastlab - 09-05-2026
That matches what I was seeing. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: CPU-only fallback performance for Gemma - routearch - 09-05-2026
Interesting, I hadn't tried that. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: CPU-only fallback performance for Gemma - torchsmith25 - 09-05-2026
following this because I'm in the same boat. The logs were somehow calmer than I was.
Re: CPU-only fallback performance for Gemma - thetapath - 09-05-2026
Quote:following this because I'm in the same boat. The logs were somehow calmer than I was.
worth checking the driver first.
Re: CPU-only fallback performance for Gemma - dockriver - 09-05-2026
I'd approach it in a boring order. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: CPU-only fallback performance for Gemma - vectorfield - 09-05-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Once you know the plain path works, then add the faster path back in piece by piece.
Re: CPU-only fallback performance for Gemma - northgrid33 - 09-05-2026
That matches what I was seeing. The pattern around NVFP4 has been pretty similar for me.
Re: CPU-only fallback performance for Gemma - mangohouse - 09-06-2026
Can you fill in a bit more context? Are you testing this on 7900 XTX or something else?