+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: do small edge boxes make sense with llama.cpp (/showthread.php?tid=1176)
Pages:
1
2
do small edge boxes make sense with llama.cpp - northgarden37 - 10-14-2026
I'm trying to sanity-check my setup.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling batch size.
Not looking for the perfect answer, mostly what worked in real use.
Re: do small edge boxes make sense with llama.cpp - vramridge - 10-15-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.
Re: do small edge boxes make sense with llama.cpp - audiorunner74 - 10-16-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom.
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: do small edge boxes make sense with llama.cpp - llmtrack26 - 10-18-2026
Appreciate the detail. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: do small edge boxes make sense with llama.cpp - sysstation69 - 10-18-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: do small edge boxes make sense with llama.cpp - torchsmith25 - 10-18-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving.
Re: do small edge boxes make sense with llama.cpp - ggufvault - 10-18-2026
that fixed it for me too.
Re: do small edge boxes make sense with llama.cpp - minthall - 10-18-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: do small edge boxes make sense with llama.cpp - amberrunner - 10-18-2026
Did a small comparison this afternoon. On 7900 XTX with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: do small edge boxes make sense with llama.cpp - thetapath - 10-18-2026
Good point. The pattern around continuous batching has been pretty similar for me.