best sandbox for trying risky tools: anyone testing this?

[cozyarch]

Maybe I'm missing something obvious.

I'm mostly working with Mixtral in SGLang on GB200. The part I'm trying to understand is small local stacks.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.

[quantghost]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

I'm happy to test one more variant if you want a second data point.

[homelabhenry]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?