+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: tokens/sec tuning ideas (/showthread.php?tid=617)
Pages:
1
2
Re: tokens/sec tuning ideas - llm_wren - 10-06-2026
subscribed, curious where this lands.
Re: tokens/sec tuning ideas - cacheline - 10-06-2026
Quote:subscribed, curious where this lands.
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: tokens/sec tuning ideas - deltastation - 10-07-2026
Quote:Quote:subscribed, curious where this lands.
Appreciate the detail. That matches what I saw on A6000.
Re: tokens/sec tuning ideas - mintforge - 10-10-2026
What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: tokens/sec tuning ideas - lablab - 10-10-2026
My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: tokens/sec tuning ideas - vectorpoint - 10-10-2026
That matches what I was seeing. I ended up in about the same place using TensorRT-LLM.