+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: tokens/sec tuning ideas: worth it? (/showthread.php?tid=572)
Pages:
1
2
tokens/sec tuning ideas: worth it? - macronest - 09-29-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Mixtral, SGLang, A6000, with a lot of attention on consumer GPUs.
Feel free to tear it apart if you see something dumb.
Re: tokens/sec tuning ideas: worth it? - northhall88 - 09-30-2026
Did a small comparison this afternoon. On mini PC with vLLM, I saw basically the same behavior.
Re: tokens/sec tuning ideas: worth it? - marbletrack - 10-01-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: tokens/sec tuning ideas: worth it? - cozygrid12 - 10-01-2026
Good point. That matches what I saw on B200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. The logs were somehow calmer than I was.
Re: tokens/sec tuning ideas: worth it? - vectorhouse71 - 10-02-2026
I hit almost the same issue last night.
Re: tokens/sec tuning ideas: worth it? - northgarden37 - 10-02-2026
That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: tokens/sec tuning ideas: worth it? - northgrid33 - 10-03-2026
What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: tokens/sec tuning ideas: worth it? - cachelab - 10-03-2026
That matches what I was seeing. That matches what I saw on GB200.
Re: tokens/sec tuning ideas: worth it? - macrotinker - 10-04-2026
Quote:That matches what I was seeing. That matches what I saw on GB200.
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: tokens/sec tuning ideas: worth it? - sparrowghost - 10-04-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.