Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
prefill/decode separation in practice with PyTorch
#2
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
measure twice, benchmark three times
Reply


Messages In This Thread
Re: prefill/decode separation in practice with PyTorch - by cacheline - 08-24-2026, 09:43 PM

Forum Jump:


Users browsing this thread: 1 Guest(s)