continuous batching wins and losses with SGLang

[thetashed24]

A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.