+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: working config for 32k context with SGLang (/showthread.php?tid=616)
working config for 32k context with SGLang - sparrowhall - 10-01-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with prefill/decode split.
Maybe obvious to some of you, but it clicked for me this week.
Re: working config for 32k context with SGLang - kernelgrid - 10-01-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: working config for 32k context with SGLang - rocmriley - 10-01-2026
following this because I'm in the same boat.
Re: working config for 32k context with SGLang - rasterbench - 10-01-2026
I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: working config for 32k context with SGLang - vramstream - 10-01-2026
Quote:I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: working config for 32k context with SGLang - mangoquill - 10-01-2026
A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: working config for 32k context with SGLang - cachebench - 10-02-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.