dual GPU split inference for Mistral

[paperrunner]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.

[turbogrid11]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[cudapoint]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchwire]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.

[turboyard94]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[dockkite69]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[rocmriley]

Good point. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchwire]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching.

[signaltrack44]

Need a little more detail here. Is this local-only or are you routing anything externally?

[amberhall]

worth checking the driver first.