tokens/sec tuning ideas

[latentgarden]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: Ollama on 7900 XTX, keeping INT4 in mind from the start.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[dockkite69]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[cobalttinker]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[quiettoken]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[visionforge]

worth checking the driver first.

[fusedriver]

My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[visionforge]

I hit almost the same issue last night.

[labline]

Appreciate the detail. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[fusedriver]

that fixed it for me too.

[oxideworks59]

that fixed it for me too.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.