driver/framework news roundup

[nanogarden]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, M3 Max, with a lot of attention on tail latency.

Feel free to tear it apart if you see something dumb.

[deepseeked]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[southpatch]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[bytepilot]

that fixed it for me too.

[circuitkite]

I hit almost the same issue last night.

[nanogarden]

I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.

[dockriver]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[amberrunner]

A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?

[mangopeak72]

following this because I'm in the same boat.