+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: NVIDIA / CUDA (https://www.datasculpting.ai/inference/forumdisplay.php?fid=31)
+--- Thread: batching changes after update with llama.cpp (/showthread.php?tid=650)
batching changes after update with llama.cpp - dockember - 10-02-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.
Wondering if this is just where the whole space is heading now.
Re: batching changes after update with llama.cpp - thetapath - 10-02-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs.
Re: batching changes after update with llama.cpp - audioriver - 10-02-2026
Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: batching changes after update with llama.cpp - latentgarden - 10-02-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: batching changes after update with llama.cpp - audiopath66 - 10-02-2026
What worked for me was pretty simple. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: batching changes after update with llama.cpp - latentgarden - 10-03-2026
following this because I'm in the same boat.
Re: batching changes after update with llama.cpp - orbitbench6 - 10-04-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.