serving on Blackwell

[kernelgrid]

Did a bit of testing and thought I'd post the result here.

I tried this with Kimi using SGLang on Threadripper box. The headline result was better than I expected, but embeddings still ended up being the limiting factor.

Curious if anyone else landed in the same place.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[macrotinker]

I don't think that's the whole story. I've seen people blame NVFP4 when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[rocmriley]

Interesting, I hadn't tried that. That matches what I saw on Threadripper box. The biggest change for me was simplifying the stack instead of adding one more optimization layer.