+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: tokens/sec tuning ideas (/showthread.php?tid=617)
Pages:
1
2
tokens/sec tuning ideas - latentgarden - 10-01-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: Ollama on 7900 XTX, keeping INT4 in mind from the start.
Feel free to tear it apart if you see something dumb.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: tokens/sec tuning ideas - dockkite69 - 10-01-2026
My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: tokens/sec tuning ideas - cobalttinker - 10-01-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: tokens/sec tuning ideas - quiettoken - 10-02-2026
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: tokens/sec tuning ideas - visionforge - 10-02-2026
worth checking the driver first.
Re: tokens/sec tuning ideas - fusedriver - 10-03-2026
My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: tokens/sec tuning ideas - visionforge - 10-03-2026
I hit almost the same issue last night.
Re: tokens/sec tuning ideas - labline - 10-03-2026
Appreciate the detail. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: tokens/sec tuning ideas - fusedriver - 10-04-2026
that fixed it for me too.
Re: tokens/sec tuning ideas - oxideworks59 - 10-06-2026
Quote:that fixed it for me too.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.