continuous batching wins and losses with Ollama

[amberrunner]

I don't think that's the whole story. I've seen people blame tail latency when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

If you post your exact settings, people here can probably narrow it down pretty fast.