batching changes after update with llama.cpp

[dockember]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Wondering if this is just where the whole space is heading now.

[thetapath]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs.

[audioriver]

Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[latentgarden]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[audiopath66]

What worked for me was pretty simple. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[latentgarden]

following this because I'm in the same boat.

[orbitbench6]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.