working config for 32k context

[relayforge]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.

[batchshed]

Posting this in case it saves someone else a little time.

I hit almost the same issue last night.

[vectorember]

Posting this in case it saves someone else a little time.

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[llmpeak]

Good point. That matches what I saw on 7900 XTX.

[relayforge]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[vramstream]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

[labnest]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[vectorport]

Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[phaseport36]

Maybe, but I had the opposite result. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[kernelgrid]

Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.