+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: Mobile (https://www.datasculpting.ai/inference/forumdisplay.php?fid=73)
+--- Thread: on-device inference worth the effort with llama.cpp (/showthread.php?tid=1411)
on-device inference worth the effort with llama.cpp - marbleyard32 - 10-17-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and A6000.
Simple and boring is a valid answer.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
Re: on-device inference worth the effort with llama.cpp - marbleyard32 - 10-17-2026
Quote:Posting this in case it saves someone else a little time.
Tried to reproduce it just now. On B200 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: on-device inference worth the effort with llama.cpp - cozysmith - 10-17-2026
What worked for me was pretty simple. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: on-device inference worth the effort with llama.cpp - sparrowghost - 10-18-2026
Good point. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
I'm happy to test one more variant if you want a second data point.
Re: on-device inference worth the effort with llama.cpp - llmpeak - 10-18-2026
I hit almost the same issue last night.
Re: on-device inference worth the effort with llama.cpp - turbosupply83 - 10-18-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of the
A couple questions before I guess too hard. Are you testing this on 7900 XTX or something else?
Re: on-device inference worth the effort with llama.cpp - sparrowshed - 10-18-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.
Re: on-device inference worth the effort with llama.cpp - cozywire - 10-18-2026
Quote:Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: on-device inference worth the effort with llama.cpp - mangopath4 - 10-18-2026
Good point. The pattern around VRAM headroom has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: on-device inference worth the effort with llama.cpp - batchpeak - 10-19-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.