on-device inference worth the effort with llama.cpp

[marbleyard32]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and A6000.

Simple and boring is a valid answer.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.