+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: tokens/sec tuning ideas (/showthread.php?tid=468)
tokens/sec tuning ideas - rocmriley - 09-25-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like batch size decides everything downstream.
Curious whether that lines up with what other people are seeing.
Re: tokens/sec tuning ideas - oxidemixer - 09-27-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: tokens/sec tuning ideas - thetarunner - 09-30-2026
Quote:Quote:This keeps coming up in conversation so I figured I'd start a thread.
I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: tokens/sec tuning ideas - cozyarch - 09-30-2026
that fixed it for me too.
Re: tokens/sec tuning ideas - macroforge - 10-01-2026
Quote:that fixed it for me too.
Appreciate the detail. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: tokens/sec tuning ideas - labnest - 10-01-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: tokens/sec tuning ideas - vramridge - 10-04-2026
Good point. That matches what I saw on mini PC.