tokens/sec tuning ideas

[llm_wren]

subscribed, curious where this lands.

[cacheline]

subscribed, curious where this lands.

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[deltastation]

subscribed, curious where this lands.

Appreciate the detail. That matches what I saw on A6000.

[mintforge]

What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.

[lablab]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[vectorpoint]

That matches what I was seeing. I ended up in about the same place using TensorRT-LLM.