Tensor cores actually helping here with SGLang

[amberpatch25]

I'm trying to sanity-check my setup.

My current setup is SGLang with B200, and I keep running into long context.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

[phaseport36]

I'm trying to sanity-check my setup.

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[kernelghost]

I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[audiopath66]

I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not gla

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[kernelmike]

[quote]I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

My first guess would be this: On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[batchshed]

Thanks, that helps. I ended up in about the same place using SGLang.

[latentgarden]

Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cobaltspark]

Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the wo

Interesting, I hadn't tried that. That matches what I saw on mini PC. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mintforge]

[quote]Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether

I don't think that's the whole story. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

[dockember]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.