small win thread for Mixtral

[routecraft]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: Ollama on B200, keeping tool calling in mind from the start.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.

[quiettoken]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[tensorkite]

subscribed, curious where this lands. I keep promising myself this is the last rebuild.

[quiettoken]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmrunner40]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[ggufwire22]

Ran a quick check on my box. On B200 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[quiettoken]

Ran a quick check on my box. On B200 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[ggufwire22]

[quote]Ran a quick check on my box. On B200 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it d

I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.