throughput per watt reality check with llama.cpp

[griffinloom66]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[batchwire]

following this because I'm in the same boat.

[macronest]

following this because I'm in the same boat.

I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[turboscope]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[homelabhenry]

I think the bottleneck is somewhere else. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[thetafield]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[forgemixer]

Need a little more detail here. Is this local-only or are you routing anything externally?

[emberpeak]

Maybe, but I had the opposite result. I've seen people blame quant drift when the real issue was much more mundane.

[bytegarden73]

that fixed it for me too.