simple local serving recipe: early impressions

[mlxgarden]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Curious whether that lines up with what other people are seeing.

[mintstation99]

A couple questions before I guess too hard. Are you testing this on A6000 or something else?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[mlxgarden]

Need a little more detail here. Are you testing this on 4090 or something else?

[vramstream]

Need a little more detail here. Are you testing this on 4090 or something else?

Good point. I don't think your conclusion is far off.

[cachebench]

subscribed, curious where this lands.

[mlxgarden]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[relayforge]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.

[rocmhouse90]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[audionode8]

That matches what I was seeing. I don't think your conclusion is far off.

[deltagrid]

Thanks, that helps. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.