+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: throughput per watt reality check with llama.cpp (/showthread.php?tid=195)
throughput per watt reality check with llama.cpp - griffinloom66 - 09-09-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.
Maybe obvious to some of you, but it clicked for me this week.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: throughput per watt reality check with llama.cpp - batchwire - 09-09-2026
following this because I'm in the same boat.
Re: throughput per watt reality check with llama.cpp - macronest - 09-09-2026
Quote:following this because I'm in the same boat.
I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: throughput per watt reality check with llama.cpp - turboscope - 09-09-2026
Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: throughput per watt reality check with llama.cpp - homelabhenry - 09-09-2026
I think the bottleneck is somewhere else. I get why that seems right, but on GB200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: throughput per watt reality check with llama.cpp - thetafield - 09-10-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: throughput per watt reality check with llama.cpp - forgemixer - 09-11-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: throughput per watt reality check with llama.cpp - emberpeak - 09-11-2026
Maybe, but I had the opposite result. I've seen people blame quant drift when the real issue was much more mundane.
Re: throughput per watt reality check with llama.cpp - bytegarden73 - 09-11-2026
that fixed it for me too.