do small edge boxes make sense with llama.cpp

[northgarden37]

I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling batch size.

Not looking for the perfect answer, mostly what worked in real use.

[vramridge]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.

[audiorunner74]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

If you post your exact settings, people here can probably narrow it down pretty fast.

[llmtrack26]

Appreciate the detail. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[sysstation69]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[torchsmith25]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving.

[ggufvault]

that fixed it for me too.

[minthall]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[amberrunner]

Did a small comparison this afternoon. On 7900 XTX with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[thetapath]

Good point. The pattern around continuous batching has been pretty similar for me.