+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: throughput per watt reality check for Mistral (/showthread.php?tid=1051)
Pages:
1
2
throughput per watt reality check for Mistral - routecrate - 10-11-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of SGLang and 4090 felt surprisingly usable once I tuned around tool calling.
If someone wants the exact settings I can clean them up and post them.
Re: throughput per watt reality check for Mistral - visionforge - 10-11-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time.
Re: throughput per watt reality check for Mistral - llmpeak - 10-11-2026
following this because I'm in the same boat.
Re: throughput per watt reality check for Mistral - routeline23 - 10-11-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: throughput per watt reality check for Mistral - papermixer4 - 10-11-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
Re: throughput per watt reality check for Mistral - rocmplane - 10-12-2026
Appreciate the detail. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: throughput per watt reality check for Mistral - edgecoast - 10-12-2026
That matches what I was seeing. The pattern around INT4 has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: throughput per watt reality check for Mistral - bytelab - 10-12-2026
Quote:That matches what I was seeing. The pattern around INT4 has been pretty similar for me. Worth rechecking your defaults though, because one bad assumpt
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: throughput per watt reality check for Mistral - quiettoken - 10-14-2026
Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: throughput per watt reality check for Mistral - vramarch - 10-14-2026
That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.