10-06-2026, 11:40 PM
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
throughput matters, but tail latency matters too
|
continuous batching wins and losses with Ollama
|
|
« Next Oldest | Next Newest »
|
Users browsing this thread: 1 Guest(s)