|
simple local serving recipe on GB200
|
10-19-2026, 12:00 AM
Maybe, but I had the opposite result. I've seen people blame KV cache pressure when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
I'm happy to test one more variant if you want a second data point.
throughput matters, but tail latency matters too
|
|
« Next Oldest | Next Newest »
|
| Messages In This Thread |
|
simple local serving recipe on GB200 - by mlxgarden - 10-13-2026, 11:39 AM
Re: simple local serving recipe on GB200 - by sparrowghost - 10-13-2026, 02:09 PM
Re: simple local serving recipe on GB200 - by rocmriley - 10-14-2026, 02:29 PM
Re: simple local serving recipe on GB200 - by batchshed - 10-15-2026, 02:39 PM
Re: simple local serving recipe on GB200 - by modgarden - 10-15-2026, 07:11 PM
Re: simple local serving recipe on GB200 - by batchscope72 - 10-15-2026, 11:50 PM
Re: simple local serving recipe on GB200 - by audionode8 - 10-18-2026, 11:12 PM
Re: simple local serving recipe on GB200 - by cachebench - 10-18-2026, 10:09 PM
Re: simple local serving recipe on GB200 - by batchsize - 10-19-2026, 12:00 AM
Re: simple local serving recipe on GB200 - by deltagrid - 10-19-2026, 12:46 AM
Re: simple local serving recipe on GB200 - by mangoquill - 10-19-2026, 04:48 AM
|
