JAX for inference only on dual 3090

[tinycontext]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.