continuous batching wins and losses with SGLang

[cacheline]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Curious whether that lines up with what other people are seeing.

[quantyard]

I hit almost the same issue last night.

[thetaplane]

That matches what I was seeing. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[audioforge]

What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

I'm happy to test one more variant if you want a second data point.

[rocmrunner40]

I hit almost the same issue last night.

Appreciate the detail. That matches what I saw on B200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[batchsize]

Interesting, I hadn't tried that. The pattern around small local stacks has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cozykite]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.

[phasestream53]

A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetashed24]

A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.