SBC inference experiments on 4090

[byteyard]

Did a bit of testing and thought I'd post the result here.

After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.

Happy to post more details if anyone wants them. The logs were somehow calmer than I was.