Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
prefill/decode separation in practice with Ollama
#2
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
throughput matters, but tail latency matters too
Reply


Messages In This Thread
Re: prefill/decode separation in practice with Ollama - by batchsize - 09-13-2026, 02:03 AM

Forum Jump:


Users browsing this thread: 1 Guest(s)