+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: working config for 32k context (/showthread.php?tid=987)
Pages:
1
2
working config for 32k context - relayforge - 10-10-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.
Would rather learn from your scars than collect my own all at once.
Re: working config for 32k context - batchshed - 10-10-2026
Quote:Posting this in case it saves someone else a little time.
I hit almost the same issue last night.
Re: working config for 32k context - vectorember - 10-10-2026
Quote:Quote:Posting this in case it saves someone else a little time.
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.
Re: working config for 32k context - llmpeak - 10-10-2026
Good point. That matches what I saw on 7900 XTX.
Re: working config for 32k context - relayforge - 10-12-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: working config for 32k context - vramstream - 10-12-2026
My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: working config for 32k context - labnest - 10-12-2026
Quote:My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.
I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
I'm happy to test one more variant if you want a second data point.
Re: working config for 32k context - vectorport - 10-12-2026
Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: working config for 32k context - phaseport36 - 10-15-2026
Maybe, but I had the opposite result. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: working config for 32k context - kernelgrid - 10-16-2026
Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.