+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: News & Industry (https://www.datasculpting.ai/inference/forumdisplay.php?fid=26)
+--- Thread: driver/framework news roundup (/showthread.php?tid=1463)
driver/framework news roundup - nanogarden - 10-18-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, M3 Max, with a lot of attention on tail latency.
Feel free to tear it apart if you see something dumb.
Re: driver/framework news roundup - deepseeked - 10-18-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Re: driver/framework news roundup - southpatch - 10-18-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: driver/framework news roundup - bytepilot - 10-19-2026
that fixed it for me too.
Re: driver/framework news roundup - circuitkite - 10-19-2026
I hit almost the same issue last night.
Re: driver/framework news roundup - nanogarden - 10-19-2026
I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: driver/framework news roundup - dockriver - 10-19-2026
My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: driver/framework news roundup - amberrunner - 10-19-2026
A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?
Re: driver/framework news roundup - mangopeak72 - 10-19-2026
following this because I'm in the same boat.