+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: batching changes after update for Llama (/showthread.php?tid=816)
Pages:
1
2
batching changes after update for Llama - kernelmike - 10-06-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between MLX and llama.cpp for Kimi. The interesting difference for me has been prefill/decode split.
What would you pick if you had to live with it for a few months?
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: batching changes after update for Llama - kernelmike - 10-07-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: batching changes after update for Llama - thetapath - 10-07-2026
Interesting, I hadn't tried that. The pattern around quant drift has been pretty similar for me.
Re: batching changes after update for Llama - audioriver - 10-09-2026
Quote:Interesting, I hadn't tried that. The pattern around quant drift has been pretty similar for me.
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: batching changes after update for Llama - thetapath - 10-09-2026
A couple questions before I guess too hard. Are you testing this on mini PC or something else?
Re: batching changes after update for Llama - amberpatch25 - 10-10-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: batching changes after update for Llama - edgecoast - 10-11-2026
That matches what I was seeing. That matches what I saw on M3 Max. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: batching changes after update for Llama - kernelmike - 10-11-2026
Quote:That matches what I was seeing. That matches what I saw on M3 Max. Worth rechecking your defaults though, because one bad assumption can make the rest
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: batching changes after update for Llama - edgecoast - 10-12-2026
My first guess would be this: On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: batching changes after update for Llama - dockriver - 10-15-2026
Quote:My first guess would be this: On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.
That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.