continuous batching wins and losses with Ollama

[pixelcrate82]

Good point. I ended up in about the same place using SGLang.

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.