serving on Blackwell: what are you seeing?

[northcraft90]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and Ollama for Llama. The interesting difference for me has been MoE serving.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it.

[torchtrack24]

Spent some time on this today and wanted to compare notes.

Appreciate the detail. I ended up in about the same place using llama.cpp. I keep promising myself this is the last rebuild.