dual GPU split inference for Mistral

[circuitwire58]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[dockkite69]

Thanks, that helps. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer. The logs were somehow calmer than I was.

[vrambandit]

same here on my side.

[rocmriley]

same here on my side.

same here on my side.

[paperrunner]

same here on my side.

That matches what I was seeing. The pattern around prefix caching has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.