+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: tokens/sec tuning ideas (/showthread.php?tid=130)
Pages:
1
2
tokens/sec tuning ideas - orbitgarden - 09-04-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.
Curious whether that lines up with what other people are seeing. Turns out 'just one more test' was a lie.
Re: tokens/sec tuning ideas - byteyard - 09-05-2026
A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: tokens/sec tuning ideas - promptfield - 09-05-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: tokens/sec tuning ideas - kernelghost - 09-06-2026
Quote:Need a little more detail here. Is this local-only or are you routing anything externally?
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: tokens/sec tuning ideas - vrambandit - 09-06-2026
Thanks, that helps. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: tokens/sec tuning ideas - macrostack69 - 09-06-2026
Appreciate the detail. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: tokens/sec tuning ideas - vectorfield - 09-06-2026
I'd start with the simpler stack.
Re: tokens/sec tuning ideas - oxidegrove40 - 09-07-2026
I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: tokens/sec tuning ideas - vrambandit - 09-07-2026
Need a little more detail here. Are you testing this on Threadripper box or something else?
Re: tokens/sec tuning ideas - bytelab - 09-07-2026
Quote:Need a little more detail here. Are you testing this on Threadripper box or something else?
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.