|
prefill/decode separation in practice on B200
|
10-18-2026, 03:34 PM
My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
throughput matters, but tail latency matters too
|
|
« Next Oldest | Next Newest »
|
| Messages In This Thread |
|
prefill/decode separation in practice on B200 - by cacheline - 10-15-2026, 12:34 AM
Re: prefill/decode separation in practice on B200 - by thetashed24 - 10-15-2026, 12:37 PM
Re: prefill/decode separation in practice on B200 - by signalcraft14 - 10-18-2026, 12:42 PM
Re: prefill/decode separation in practice on B200 - by vrambandit - 10-18-2026, 12:57 PM
Re: prefill/decode separation in practice on B200 - by amberrunner - 10-18-2026, 02:28 PM
Re: prefill/decode separation in practice on B200 - by batchsize - 10-18-2026, 03:34 PM
Re: prefill/decode separation in practice on B200 - by batchember - 10-18-2026, 04:10 PM
|
