+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: do small edge boxes make sense with llama.cpp (/showthread.php?tid=932)
Pages:
1
2
do small edge boxes make sense with llama.cpp - kernelghost - 10-08-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.
Happy to post more details if anyone wants them.
This is mostly for local work rather than public serving.
Re: do small edge boxes make sense with llama.cpp - sanepeak48 - 10-08-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed. At this point the real bottleneck is my patience.
Re: do small edge boxes make sense with llama.cpp - cobaltworks25 - 10-08-2026
That matches what I was seeing. The pattern around prefill/decode split has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: do small edge boxes make sense with llama.cpp - tensorkite - 10-08-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.
Re: do small edge boxes make sense with llama.cpp - amberpatch25 - 10-08-2026
Quote:I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.
Thanks, that helps. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: do small edge boxes make sense with llama.cpp - visionforge - 10-11-2026
I think the bottleneck is somewhere else. I've seen people blame VRAM headroom when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: do small edge boxes make sense with llama.cpp - tensorfield88 - 10-12-2026
Quote:I think the bottleneck is somewhere else. I've seen people blame VRAM headroom when the real issue was much more mundane. Worth testing the simpler ex
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. At this point the real bottleneck is my patience.
Re: do small edge boxes make sense with llama.cpp - flintriver50 - 10-12-2026
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: do small edge boxes make sense with llama.cpp - audiorunner74 - 10-12-2026
A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: do small edge boxes make sense with llama.cpp - tensorport - 10-15-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there.