JSON mode across models for Mistral

[pixelcrate82]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Simple and boring is a valid answer.

[mangopath4]

Thanks, that helps. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[oxideworks59]

My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[tensorfield88]

Thanks, that helps. I ended up in about the same place using PyTorch.

[llmarch51]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[cacheplane98]

Good point. The pattern around workstation thermals has been pretty similar for me.

[emberquill]

that fixed it for me too.

[vramstream]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[quietsupply11]

Can you fill in a bit more context? Are you testing this on dual 3090 or something else?

[sparrowgrid]

Can you fill in a bit more context? Are you testing this on dual 3090 or something else?

I'd start with the simpler stack.