on-device inference worth the effort with llama.cpp

[marbleyard32]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and A6000.

Simple and boring is a valid answer.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[marbleyard32]

Posting this in case it saves someone else a little time.

Tried to reproduce it just now. On B200 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[cozysmith]

What worked for me was pretty simple. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[sparrowghost]

Good point. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

I'm happy to test one more variant if you want a second data point.

[llmpeak]

I hit almost the same issue last night.

[turbosupply83]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of the

A couple questions before I guess too hard. Are you testing this on 7900 XTX or something else?

[sparrowshed]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

[cozywire]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

Need a little more detail here. What model, what runtime, and roughly what context size?

[mangopath4]

Good point. The pattern around VRAM headroom has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[batchpeak]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.