working config for 32k context

[mintstation99]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like tokens/sec decides everything downstream.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[kernelgrid]

Appreciate the detail. The pattern around tail latency has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rocmriley]

Appreciate the detail. The pattern around tail latency has been pretty similar for me. Worth rechecking your defaults though, because one bad assumpti

What worked for me was pretty simple. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[cachebench]

same here on my side.

[llmpeak]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[deltagrid]

I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[rocmriley]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[modgarden]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[visionforge]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[kernelgrid]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.