+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: working config for 32k context with SGLang (/showthread.php?tid=1025)
working config for 32k context with SGLang - visionforge - 10-10-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between Ollama and SGLang for Llama. The interesting difference for me has been tokens/sec.
Interested in tradeoffs more than fan loyalty.
This is mostly for local work rather than public serving.
Re: working config for 32k context with SGLang - rocmhouse90 - 10-10-2026
Quote:Spent some time on this today and wanted to compare notes.
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: working config for 32k context with SGLang - audioriver - 10-11-2026
that fixed it for me too.
Re: working config for 32k context with SGLang - llmpeak - 10-11-2026
same here on my side.
Re: working config for 32k context with SGLang - audioriver - 10-11-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: working config for 32k context with SGLang - sparrowbrook - 10-11-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: working config for 32k context with SGLang - llmpeak - 10-11-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints.
Re: working config for 32k context with SGLang - modgarden - 10-11-2026
What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: working config for 32k context with SGLang - rocmriley - 10-12-2026
I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time. I keep promising myself this is the last rebuild.
Re: working config for 32k context with SGLang - audionode8 - 10-12-2026
Maybe, but I had the opposite result. I get why that seems right, but on mini PC the bottleneck felt elsewhere. At this point the real bottleneck is my patience.