+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Prompting & Context Engineering (https://www.datasculpting.ai/inference/forumdisplay.php?fid=15)
+--- Forum: Structured Output (https://www.datasculpting.ai/inference/forumdisplay.php?fid=63)
+--- Thread: how strict do you make schemas for Mistral (/showthread.php?tid=1183)
Pages:
1
2
how strict do you make schemas for Mistral - nanobrook - 10-13-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Phi using vLLM on 7900 XTX. The headline result was better than I expected, but tail latency still ended up being the limiting factor.
Happy to post more details if anyone wants them.
This is mostly for local work rather than public serving.
Re: how strict do you make schemas for Mistral - rocmcrate26 - 10-14-2026
Quote:Did a bit of testing and thought I'd post the result here.
I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: how strict do you make schemas for Mistral - routecrate - 10-14-2026
Quote:Quote:Did a bit of testing and thought I'd post the result here.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split.
Re: how strict do you make schemas for Mistral - relayforge - 10-14-2026
Thanks, that helps. I ended up in about the same place using vLLM.
Re: how strict do you make schemas for Mistral - mangoquill - 10-14-2026
What exact setup are you running? Are you testing this on M3 Max or something else?
Re: how strict do you make schemas for Mistral - sparrowcraft - 10-16-2026
Quote:What exact setup are you running? Are you testing this on M3 Max or something else?
Tried to reproduce it just now. On dual 3090 with PyTorch, I saw basically the same behavior.
Re: how strict do you make schemas for Mistral - visionforge - 10-16-2026
I'd start with the simpler stack.
Re: how strict do you make schemas for Mistral - mintstation99 - 10-18-2026
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: how strict do you make schemas for Mistral - macrostack69 - 10-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. Not glamorous advice, but repeatable baselines save a lot of time.
Re: how strict do you make schemas for Mistral - turboarch95 - 10-18-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.