throughput per watt reality check for Mistral

[routecrate]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of SGLang and 4090 felt surprisingly usable once I tuned around tool calling.

If someone wants the exact settings I can clean them up and post them.

[visionforge]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time.

[llmpeak]

following this because I'm in the same boat.

[routeline23]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[papermixer4]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[rocmplane]

Appreciate the detail. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[edgecoast]

That matches what I was seeing. The pattern around INT4 has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[bytelab]

That matches what I was seeing. The pattern around INT4 has been pretty similar for me. Worth rechecking your defaults though, because one bad assumpt

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[quiettoken]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vramarch]

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.