how strict do you make schemas for Mistral

[nanobrook]

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using vLLM on 7900 XTX. The headline result was better than I expected, but tail latency still ended up being the limiting factor.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

[rocmcrate26]

Did a bit of testing and thought I'd post the result here.

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[routecrate]

Did a bit of testing and thought I'd post the result here.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split.

[relayforge]

Thanks, that helps. I ended up in about the same place using vLLM.

[mangoquill]

What exact setup are you running? Are you testing this on M3 Max or something else?

[sparrowcraft]

What exact setup are you running? Are you testing this on M3 Max or something else?

Tried to reproduce it just now. On dual 3090 with PyTorch, I saw basically the same behavior.

[visionforge]

I'd start with the simpler stack.

[mintstation99]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[macrostack69]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. Not glamorous advice, but repeatable baselines save a lot of time.

[turboarch95]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.