JAX for inference only

[quietarch]

Did a bit of testing and thought I'd post the result here.

Short version: MLX behaved well for me, but only after I stopped ignoring quant drift.

Curious if anyone else landed in the same place.

[plainscope50]

Did a bit of testing and thought I'd post the result here.

that fixed it for me too.

[vramridge]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[bytestack]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.

[cacheworks]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[northhall88]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[lablab]

What exact setup are you running? Is this local-only or are you routing anything externally?

[deepseeked]

same here on my side.

[modbench]

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.

[dockkite69]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?