+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: General (https://www.datasculpting.ai/inference/forumdisplay.php?fid=25)
+--- Thread: small win thread for Mixtral (/showthread.php?tid=31)
small win thread for Mixtral - routecraft - 08-26-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: Ollama on B200, keeping tool calling in mind from the start.
If someone wants the exact settings I can clean them up and post them.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: small win thread for Mixtral - quiettoken - 08-28-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: small win thread for Mixtral - tensorkite - 08-28-2026
subscribed, curious where this lands. I keep promising myself this is the last rebuild.
Re: small win thread for Mixtral - quiettoken - 08-28-2026
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: small win thread for Mixtral - rocmrunner40 - 08-28-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: small win thread for Mixtral - ggufwire22 - 08-31-2026
Ran a quick check on my box. On B200 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: small win thread for Mixtral - quiettoken - 08-31-2026
Quote:Ran a quick check on my box. On B200 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: small win thread for Mixtral - ggufwire22 - 08-31-2026
Quote:[quote]Ran a quick check on my box. On B200 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it d
I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.