+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: working config for 32k context (/showthread.php?tid=722)
Pages:
1
2
working config for 32k context - mintstation99 - 10-04-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like tokens/sec decides everything downstream.
Curious whether that lines up with what other people are seeing.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: working config for 32k context - kernelgrid - 10-05-2026
Appreciate the detail. The pattern around tail latency has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: working config for 32k context - rocmriley - 10-05-2026
Quote:Appreciate the detail. The pattern around tail latency has been pretty similar for me. Worth rechecking your defaults though, because one bad assumpti
What worked for me was pretty simple. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: working config for 32k context - cachebench - 10-05-2026
same here on my side.
Re: working config for 32k context - llmpeak - 10-05-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: working config for 32k context - deltagrid - 10-05-2026
I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: working config for 32k context - rocmriley - 10-05-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: working config for 32k context - modgarden - 10-05-2026
Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: working config for 32k context - visionforge - 10-06-2026
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: working config for 32k context - kernelgrid - 10-06-2026
A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.