continuous batching wins and losses

[rocmrunner40]

Did a bit of testing and thought I'd post the result here.

I tried this with FLUX using vLLM on 5090. The headline result was better than I expected, but small local stacks still ended up being the limiting factor.

Curious if anyone else landed in the same place.

[signalcraft14]

That matches what I was seeing. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cacheline]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[routearch]

Appreciate the detail. The pattern around token cost has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[turboarch95]

Good point. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[phasestream53]

Maybe, but I had the opposite result. I've seen people blame reranking when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[fusedtinker]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

I'm happy to test one more variant if you want a second data point.

[batchcraft]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. Once you know the plain path works, then add the faster path back in piece by piece.

[promptpoint]

Tried to reproduce it just now. On dual 3090 with Ollama, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[nanogarden]

I hit almost the same issue last night.