M3 Max for local use

[mellowstream]

That matches what I was seeing. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding o

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.

[amberpatch25]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.